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1. D’FUSION COMPUTER VISION - INTRODUCTION 


What is D’Fusion Computer Vision by Total Immersion™ ? 


D’Fusion Computer Vision (also called D’FusionCV®) is a software package based on a patented innovative 
technology designed to retrieve “natural” information provided by various real objects found in an image or on a 
video stream. Once this natural data is collected, objects can be identified and followed (or tracked) through 
consecutive images extracted from the video stream in real-time. Experts usually call this technology “MarkerLess 
Tracking” or “Natural Features Tracking”. 


D’Fusion Computer Vision has been designed to achieve the following tasks: 


e Capture, convert and undistort video streams from video cameras and from video files. The capture 
module supports almost all camera devices such as HD/SD video cameras and web cameras. 


e Calibrate s camera in order to estimate internal camera parameters such as the focal, optical center and 
the sensor distortion. 

e Track 2D/3D natural objects when visible in the video stream. These targets can present various shapes 
and various textures (playing cards, flyers, boards, or more complex objects such as boxes, cars, natural 
scenes, broadcast studios or faces, etc.). 

e Track a plurality of objects at the same time into the video stream (Multiple Object Tracking). 

e Recognize an object in the video stream among several (Object Recognition). 

e Detect simple interactions between the user and the real object that is tracked (Pointing Detection). 

e Provide basic image processing functions such as Gesture Detection, Background Subtraction, or Face 
Detection. 

e Send information, including the position and the orientation of the targets, through the network or using 
an SDK. 


e Register new user specified targets through a Graphical User Interface and create new tracking scenarios 
in a minute. 


The D’Fusion Computer Vision package is composed of: 


e D’Fusion Studio Computer Vision Pro (objects registration tool - see section 3 for a complete 
description), designed to set up real-time MarkerLess Tracking applications. 


e The D’Fusion Pro Player Computer Vision (see section 2.4 for a complete description), the 
application that achieves MarkerLess Tracking in real-time. This tool can be used to test scenario created 
with D’Fusion Studio Computer Vision. 


e The D’Fusion Plugin Computer Vision (see section 2.5 for a complete description), that can be used 
to design complex Augmented Reality applications based on the D’Fusion Augmented Reality engine. 


What is D’Fusion Augmented Reality by Total Immersion™? 


D’Fusion Augmented Reality is a software dedicated to produce and create Augmented Reality applications. This 
powerful softwaer can be used to create complex AR scenarios using D’Fusion Computer Vision’s functionalities. 
Using this tool, the user can merge video streams, 3D effects and user interactions in order to design the most 
attractive Augmented Reality scenarios. A specific plug-in (D’FusionPluginCV) has been created in order to make 
D’Fusion CV and D’Fusion AR work on the same computer. 


What is the difference between Classic & Lite Tracking ? 


Total Immersion recently improved its tracking technology to make AR applications work on mobile (depending on 
the OS and performances) and flash platforms. As a consequence, the “Classic” Recognition/Tracking 
evolved into a “Lite” Recognition/Tracking in order to save computation time without a loss of robustness. 
This evolution is based on heavy R&D innovations and is protected by patents. 
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1.1 EXTERNAL DOCUMENTATION 
[1] D’Fusion Studio - User Guide, Total Immersion 
DFusion Studio - User Guide.pdf 
[2] D’Fusion AR - Reference Manual, Total Immersion 
DFusion AR — Reference Manual.pdf 
1.2 GLOSSARY & ACRONYMS 
TI Total Immersion 
AR Augmented Reality 
MLT Marker Less Tracking 
CV Computer Vision 
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2. RECOGNITION & TRACKING WITH D’FUSION CV 


2.1 PACKAGE INSTALLATION 


This section explains the installation of D’Fusion Computer Vision and the organization of D’Fusion Computer Vision 
software package. Notice that the user has to install the application’s main components using the 
“D’FusionMLT.exe” installation’s file or using the “index.html” provided within the installation CDROM. By default, 
the software is installed on “C:/Program Files/Total Immersion/”. 


= |) DFusionc¥ 3,20,18101 


= © Bin 
# (3 RealTimeTracker D’Fusion Plugin Computer Vision for D’Fusion Augmented Reality plugin 
(9 Redist D’Fusion Pro Player Computer Vision 
= (9 Documentation Tools to install hasp’s drivers and visual redist 
= (© Appi 
(9 ReferenceManual D’Fusion Computer Vision - Reference Manual 
( License 


Notice that a Hasp Key is provided with the D’Fusion Computer Vision package. This key must be plugged before 
launching the D’Fusion Computer Vision package’s applications. 


2.2 SCENARIO ORGANIZATION 


To be able to use the MarkerLess Tracking functionalities, you have to configure your scenario using D’Fusion 
Studio Computer Vision. In most cases, a scenario encapsulates the following elements, located on the disk: 


e An xml configuration file corresponding to the tracking scenario (usually “tracker.xml”), 

e An xml configuration file corresponding to the camera calibration (usually “cameraCalib.xml”), 

e An xml configuration file corresponding to the camera capture (usually “cameraConfig.xml”), 

e Various images corresponding to registered objects. 

e Binary files that are useful to store objects statistical structures (to handle the recognition capabilities) 
Notice that if you are a partner of Total Immersion, you can find some samples on the partner’s website. 
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2.3 KEY CONCEPTS 


This section provides the key features to understand the concepts intrinsic to D'Fusion Computer Vision. Depending 
on the targeted platform, you will have to choose between the “Classic” tracker (PC, MAC & standard 
applications) and the “Lite” tracker (phones & flash applications). 


2.3.1 MLT Process 

The MarkerLess Recognition and Tracking technology provided by D’Fusion Computer Vision is a process composed 
of several steps: recognition, initialization and tracking. 

Recognition step 


D’Fusion Computer Vision has the possibility to process a recognition step. A recognition can be done when 
a certain amount of “objects” or “targets” are defined. The recognition step allows the user to identify 
which object is present in the current image. 


Automatic Initialization step 


This is a specific initialization step that does not require the real object to be placed close to the recorded 
keyframe pose. If enabled, this mode replaces the traditional initialization step and allows the user to 
proceed with a fully automatic initialization without referring to a specific object pose. 


Face Tracking Automatic Initialization step 


During this initialization step, D’Fusion Computer Vision is capable of recognizing faces automatically and 
then following them in the video stream. 


Initialization step 


During the jnitialization step, a set of keyframes are used to estimate a first pose for the objects to be 
retrieved from the video stream. If the object is not present in the current image, the algorithm stays into 
this initialization stage. A target can be retrieved from the image if there is at least a recorded keyframe 
where the object is defined nearly at the same pose. Once a target is found, the application will proceed to 
the tracking step. 


Tracking step 


When in tracking step, an object pose is estimated into the currently captured video image according to its 
previous pose and using the keyframe information. If the object is not visible any more, the application 
returns to the /nitialization step. 


Pointing Detection step 


The pointing detection is a module provided to detect perturbations that appear over the tracked target. As 
a consequence, this functionality works only when objects are tracked in the video stream. This technology 
is not available in the “Lite” tracking mode. 
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2.3.2 Main Concepts 


2.3.2.1 Objects (Targets) 


An object to track has to be identified and followed using successive frames information. This object has to 
be associated with a corresponding geometry called a “grid” or a virtual model (.obj meshes). In case the 
user wants to track planar objects, the grid is generated automatically. 


Figure 1 — Examples of targets (planar tracking, landscape tracking, face tracking) 


2.3.2.2 Multiple Object Tracking 


The capability to track multiple objects at the same time and/or successively using the same video capture. 


tracking: = 


Figure 2 — Example of multiple-object tracking 


2.3.2.3 6 Degree of Freedom Pose 


A 6DoF pose (or simply a pose) encapsulates a 3D position and a 3D orientation in the space coordinate 
system. This is the way we will describe the real object position and orientation according to the camera 
coordinate system. 
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2.3.2.4 Camera Calibration 


The calibration associated to a given camera describes the way 3D data is projected into the 2D sensor. 
These internal camera parameters (focal, optical center and distortion) can be estimated from within 
D’Fusion Studio Computer Vision and are stored into a specific x7m/ file. 


2.3.2.5 Keyframe 


A keyframe is an important notion to understand how the MarkerLess Tracking works. It encapsulates: 
e a6 Degree of Freedom Pose (using a model view matrix), 
e a bitmap image that contains the object and, 
¢ acorresponding calibration. 


Therefore the pose of the keyframe describes the position and the orientation of the object to be tracked in 
the image according to the camera calibration. 


In D’Fusion Computer Vision there are two kinds of keyframes: the offline keyframes (specified manually 
by the user through the Keyframe Manager) and the online keyframes (created automatically by the 
D’Fusion Player CV). 


2.3.2.6 Interest Points (Feature Points, Keypoints) 


D’Fusion Computer Vision uses feature points to recognize and track objects. Feature points are detected 
near high frequency areas so that they can each describe a small part of the image. By matching these 
feature points, known images can be recognized and tracked in video frames. 


Figure 3 — Interest Points (displayed with white crosses) 
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2.3.2.7 Sub-Image Tracking 


Sub-Image Tracking is a tracking mode that can be used to recognize and follow small objects in the video 
stream. In this case, processed images are not downscaled and the tracking is carried out with a full 
resolution cropped image. 


Sub-Image (470x280) Down-Scaled image (470x280) 


Figure 4 — Example of sub-image and downscale image scenario with a 1080i capture 
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2.3.2.8 Black Box 


The Black Box feature allows you to register a large number of images to be recognized for a single object. 


This is possible thanks to the ability (enabled by the Black Box feature) to create groups of auto-init images 
for a single object. At each frame, recognition is carried out on the images of processed groups only and, 
by deciding how many groups are to be processed per frame, recognition time can truly be optimized. 


For example, the detection time for one image inside a group of 20 images, is nearly the same as the 
detection time for one image inside a group of 100 images. 


The Black Box also allows you to create dynamic groups, described below. 


The Black Box feature is available for the “Lite” tracking mode only, through objects of type “Plane Black 


Ww 


Box”. 


2.3.2.9 Black Box Dynamic Groups 


Dynamic groups are groups of auto-init images of type “Black Box” (and of this type only) that can be 
loaded dynamically while your MLT scenario is being executed (as opposed to when your project is 
launched). 


The ability to load groups dynamically requires you to have two types of scenarios: 


e An « authoring scenario », used for editing, which includes all the groups to be loaded at runtime 
and allows you to test and configure your groups. 


e A «runtime scenario », which corresponds to a dynamic version of your project, and consists of the 
main scenario to load at runtime (i.e. your scenario without the data set as dynamic), plus a separate 
folder for each group that is to be loaded during execution. 


Both scenarios are generated by the D’Fusion Studio CV Pro application: 


e The “authoring scenario” corresponds to the scenario that you edit when using D’Fusion Studio CV Pro, 
generated when creating (and saving) your project. 


e The “runtime scenario” corresponds to the final scenario, the one that will be loaded when launching 
your application, generated when saving your project if and only if one or more groups are set as dynamic. 


You can thus use the D’Fusion Studio CV Pro application to create, test and configure your groups, and 
specify which groups are to load dynamically, then use the “runtime scenario” when launching your 
application if you are actually using dynamic groups. 


The use of dynamic groups offers many advantages: 


e Dynamic groups can be located outside your AR scenario (project.dpd parent folder), and can thus 
be loaded beforehand (from the web) on demand. 


e This allows you for example to optimize the size of your updates: instead of updating an entire 
project with 100 movie posters, you can simply update the 20 posters requiring the update (and eventually 
corresponding to two different groups). 


e This advantage further allows you to optimize memory usage, by having in memory (and therefore 
used at runtime) only the images that are useful at a specific time “t’. This way you can also optimize 
image detection/identification delay. For example, for a catalogue organized in « departments » like « 
kitchen » « bedroom » « dining room », a user can answer « which department interests you? », then load 
only the 100-150 images that corresponds to the chosen department. 
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2.4 D’'FUSION PRO PLAYER COMPUTER VISION - HOW IT WORKS 


The D’Fusion Pro Player CV is the application that exploits keyframes and video streams to process real-time 
tracking. You can launch this application using the command: “./8in/Rea/TimeTracker/DFusionPlayerCv. exe 
tracking.xm/”~ According to the tracking scenario, the application may need to be adapted through the 
configuration file and using D’Fusion Studio Computer Vision Pro. 


The D’Fusion Player CV works by proceeding from the “Recognition & Initialization” step to the “Tracking” step. The 
figure below explains these steps and their inputs and outputs. 


— 


Previous Pose 
Estimation 


Recognition & 
Initialization 
Step 


Object Data 
Base 


Tracking Step 


Current Pose 
Estimation & 
Object index 


Figure 5 — The D’Fusion Pro Player CV 


Input information is displayed in red and resulting information in blue 
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2.5 HOW TO USE D’FUSION COMPUTER VISION IN A THIRD PARTY 
SOFTWARE 


You can receive data from the D’Fusion Pro Player CV using the D’Fusion Plugin Computer Vision with D’Fusion 
Augmented Reality. In case you are using a third party rendering software, you will have to use the UDP mode. 


UDP data 


” VizRT® 
D’Fusion D’Fusion AR 
Plugin CV Engine 
eee p| BrainStorm® 
~~! YourSoftware 


Figure 6 — Exploiting the resulting poses given by D’Fusion Computer Vision 


2.5.1 D’Fusion Augmented Reality 


D’Fusion Augmented Reality is a piece of software provided by Total Immersion designed to create and develop 
Augmented Reality (or Mixed Reality) applications. When using D’Fusion Computer Vision with D’Fusion 
Augmented Reality, a file named DFusionPluginCV.dll is provided to handle the communication between the two 
software. 


2.5.2 UDP Mode 


When using the UDP connection mode, data is received from the network. Notice that in case you want to use a 
single computer, it can be configured with the local host address (127.0.0.1). As described in chapter “3.7.11 
Output configuration (“Output Manager” panel)”, there are many ways to send UDP data through the network. 
Depending to its rendering software the user can choose the appropriate data format. 
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3. D’FUSION STUDIO COMPUTER VISION PRO 
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D’Fusion Studio Computer Vision Pro (D’Fusion Studio CV Pro) is a Graphical User Interface that helps designing 
tracking scenarios. 


D’Fusion Studio Computer Vision Pro will allow you to create and register new objects to be tracked. You can 
launch D’Fusion Studio CV Pro from the start menu: "Start > A// Programs > Total Immersion > DFusion Studio 
CV Pro > DFusionStudioCVPro”. 


Scenarios created with D’Fusion Studio CV Pro can be reused later by the D’Fusion Pro Player CV. D’Fusion Studio 


CV Pro will also allow you to setup internal parameters to ensure the quality of the real-time application. 


This chapter presents the D’Fusion Studio CV Pro user interface. Most of the features described here are also 
available through the D’Fusion Studio interface, “Tools” menu. See documentation [1] for more information about 
D’Fusion Studio. 


3.1 MENU BAR 


The menu bar provides access to the following operations: 


“File > New”: starts a new project from scratch 

“File > Load”, “File > Recent Scenario”: loads an existing project 

“File > Save”, “File > Save As ...”: saves the current project 

“File > Exit”: quits the application 

“View > Log Window”: displays information returned by the application 

“Mode > Classic”: activates the “Classic” tracking mode (default mode) 

“Mode >Lite”: activates the “Lite” tracking mode (for specific platforms) 

“User Level > Basic”: sets the basic user level panels (for beginners) 

“User Level > Advanced”: sets the advanced user level panels (for advanced users 
“? > Online Community”: opens the Total Immersion™ Community website 

“? > About D’Fusion Studio CV Pro”: displays information related to the product 
“? > D’Fusion CV - Reference Manual”: opens this documentation 


3.2 PANELS 


The following panels will allow you to set a tracking scenario: 
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“Video Manager”: specify the video capture (section 3.3) 


“Camera Calibration”: calibrate the camera used for the tracking (section 3.4), not available when 


using “Basic” user level 


“Video Processing”: specify the video stream that will be processed by the tracking (section 3.5), not 


available when using “Basic” user level 
“Scenario Manager”: register new objects (section 3.6) 


“Tracking Manager”: review tracking results according to the tracking parameters (section 3.7) 


“Output Manager”: specify networking data that will be sent through the network (section 3.7.11), not 


available when using “Lite” tracking mode 
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3.3 VIDEO CAPTURE CONFIGURATION (“VIDEO MANAGER” PANEL) 


This panel is dedicated to the video capture configuration. Notice that only Directshow®’s drivers are supported 
on PC. 


W D'Fusion Studio Computer Vision Ce) 


File View Mode ? 


Camera Calibration |Video Preprocessing ‘Scenario Manager Tracking Manager ‘Output Manager 


Source 

Type DirectShow 

Video File 

Bevice List Peripherique video USB 


Num Driver 


Parameters 
Width 
Height 
Pixel Format 
Rate 
Interlaced Not interlaced 
Inverted Horizontal 
Delay 
NbFrameBuffers 
Priority 
CPU 
“Saved selection mode 
Pro Target : 
Currently defined camera 
@Home Target : 
Manual selection through a list-at runtime 
Mobile Targets : 
Back camera 
Front camera 


Figure 7 - “Video Manager” panel 


According to the physical characteristics of the camera, you will have to fill in the camera parameters: 


e “Type”: to select the video capture type (amongst directShow, Video UEye, Video for Windows). 


e “Video File”: to specify a video recorded into a video file (Supported formats depend on codec that are 
installed on the user computer). 


e “Device List”: to select directly the camera amongst available devices. 

e “Num. Driver”: to select directly the driver of the requested capture device. 

e “Import”: to import a previously configured camera through a “.xml” file. 

e “Close”: to close currently used video capture. 

e “Open”: to confirm current modification. 

e “Width, Height”: to select a capture size (must be compliant with the device capabilities). 


e “Pixel Format”: to be selected amongst RGB24, BGR24, RGB32, UYVY422, GRAY8, RGB565, 
ARGB32 (chosen format must be compliant with the device capabilities). 
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e “Rate”: to specify capture rate in fps (must be compliant with the device capabilities). 


e “Interlaced”: to be selected according to the video signal. 0 for non-interlaced mode (e.g. using a web- 
camera), 1 or 2 for interlaced modes (must be compliant with the device capabilities). 


e “Inverted”: to specify image flips. 
e “Delay”: to specify video capture delay (a number of frame). This delay is not taken into account for the 


tracking application. 


e “NbFrameBuffers”: to specify the size of the buffer to record video frames. 
e “Priority”: to specify the video capture thread priority. 
e “CPU”: to specify the video capture thread CPU. 


The following tables will help you find the correct setting to capture video according to the input signal and the 


11/12/12 


camera used. 
Video | WebCamera/ | SD/NTSC/ | SD/PAL/ | SD/NTSC/ SD/PAL/ 
File USB Firewire | Firewire | Composite- | Composite- 
SVideo SVideo 
Num Driver -1 0, ... 0, ... 0, ... 0, ... 0, ... 
Capture Size (Width, Height) ‘a 640x480 720x480 | 720x576 | 640x480 768x576 
320x240 
Pixel Format - RGB32, RGB24 RGB24 RGB24 RGB24 
RGB24, 
BGR24 
Rate ‘ 30 29.97 25 29.97 25 
Interlaced ‘a 0 1or2 1or2 1or2 1or2 
Table 1 — Webcam and SD specifications 
HD/NTSC HD/PAL uEye/NTSC/ uEye/PAL/ 
USB USB 
Num Driver O, ... 0, ... 0, ... 0, ... 
Capture Size (Width, Height) 1920x1080 1920x1080 640x480 768x576 
1280x720 1280x720 
Pixel Format UYVY422, UYVY422, RGB24 RGB24 
RGB32 RGB32 RGB32 RGB32 
RGB565 RGB565 
Rate 29.97 (i), 30(p) 25 60 50 
Interlaced O(p), 1(i), 2¢i)__|_O(p), 1(i), 2(i) 0 0 
Table 2 — UEye & HD Cameras: (i) means interlaced, (p) means progressive 
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3.4 CAMERA CALIBRATION (“CAMERA CALIBRATION” PANEL) 


The “Camera Calibration” panel will help you calibrate your camera by calculating the camera focal, optical center 
and potential distortions. 


W D'Fusion Sti 


File View Mode 
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Cam..a Calibration 
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— 2 | Set To Scenario Get From Scenario a 
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Figure 8 - The “Camera Calibration” panel and its sub-panels “Camera” and “Calibration” 
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The “Calibration” sub-panel is designed to help you on the camera calibration process: 


e “Video Dedistortion”: to test the current distortion parameters with the current video stream. 


e “Load Process”: to import previously recorded configuration for the camera calibration process. You 
may notice that this x/7/ file doesn’t contain the calibration parameters of the camera but information 
related to the calibration process. 


e “Save Process”: to record the current configuration of the calibration process. 


e “Calibration Process”: to specify the type of process to use for the calibration (in D’Fusion Studio 
Computer Vision, the only choice is “Mode OpenCVv”). 


e “Print Pattern”: to print the chessboard pattern that will be use for the calibration. 
e “Width”, “Height”: to specify the real size of the chessboard (in millimetres). 


e “Nb Square On Width”, “Nb Square On Height”: to specify the number of squares of the 
chessboard. 


e “Nb Of Frame To Process”: total number of measures that will be used during the calibration process. 
e “Wait Frame Between Detections”: to wait between two measures. 


e “Limit Error To Keep Frame”: to specify the error threshold (in pixels) in order to keep only relevant 
measure . 


e “Limit Error To Keep First Frame”: to specify the error threshold (in pixels) in order to keep only 
relevant results for the first measure. 


e “Use Estimation”: use estimation to speed up the processing and to obtain more accurate results. 
e “Fix Optical Center”: to fix the optical center with specified “cx” & “cy” values. 

e “fx”, “fy”, “cx”, “cy”: to specify an estimated focal and optical center 

e “Zero Tangent Distortion”: to remove tangential distortion estimation in the calibration process. 
e “Fix Aspect Ratio”: to fix the aspect ratio of the sensor given by the “fx” & “fy” values. 


e “Acquisition Sech X & Y”: to specify the sub sampling factor of the proceeded image during the 
calibration process. In case the capture is interlaced, the “Acquisition Sech Y” value is fixed to 2. 


e “Launch”: to launch the calibration process according to specified parameters. 


e “Pause Process”: to pause the process. 
“Stop”: to stop the process. In this case, all measurements are lost. 


Once the calibration process is completed, results are stored in the “Camera” sub-panel: 


e “Max (X) (Y)”: to display image size obtained from the calibration process. 
e “Focal (X) (Y)”: to display focal values (in pixels) obtained from the calibration process. 


e “Optical Center (X) (Y)”: to display the optical center position in pixels obtained from the calibration 
process. 


e “Radial (k1) (k2)”: to display radial distortion obtained from the calibration process. 
e “Tangential (p1) (p2)” : to display tangential distortion obtained from the calibration process. 


e “Set To Scenario”: to apply these current parameters to the current MLT scenario. When applying, 
camera parameters will be accessible in the “Video Processing” panel (section 3.5). 


e “Get From Scenario”: get the parameters from the “Video Processing” panel and apply them to the 
“Camera Calibration Panel”. 


e “Import Camera”: to load camera calibration parameters obtained from a previous calibration process. 
e “Save Camera”: to record a .xm/ file that contains current calibration parameters. 
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3.5 PROCESSING (“VIDEO PROCESSING” PANEL) 


Use the “Video Processing” panel to fill camera calibration parameters and to specify the video stream to be 
processed in D’Fusion Studio CV Pro. 


W D'Fusion Studio Computer Vision er 
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Max (x, y) 
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Skip odd or even Frame 


Sub Image 


Use SubImage 


Figure 9 — “Video Processing” panel 
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The first set of edit boxes are dedicated to internal camera parameters specification: 


e “Load Calibration”: to load camera calibration parameters obtained from a previous project. 


e “Optical Center (cx, cy)”: the camera optical center (expressed in pixels according to the “Max X” & 
“Max Y” size). 


¢ “Focal (fx, fy)”: the camera focal (expressed in pixels according to the “Max X” & “Max Y” size). 
e “Max X” & “Max Y”: referential image size (in pixels) for the calibration. 

e “Radial (a1, a2)”: to display radial distortion obtained from the calibration process. 

e “Tangential (ki, k2)” : to display tangential distortion obtained from the calibration process. 

e “Activate distortion”: to enable video distortion during the tracking process. 

e Then it appears various boxes to specify the proceeded video stream properties : 

e “Processing size”: to setup the input image size. 


e “Skip odd or even Frame”: when using interlaced mode, indicates if we wish to keep only even (or 
odd) frames and skip odd (or even) ones, or if we want to use the deinterlaced odd and even frame. 
e “Use SubImage”: to specify a sub image scenario. Notice that once this check box is validated, the user 


will be able to use the mouse on the display window to crop its region of interest. This functionality is not 
provided with the “Lite” tracking. 
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3.6 SCENARIO CREATION (“SCENARIO MANAGER” PANEL) 


The “Scenario Manager” panel is dedicated to scenario creation: create new targets, specify their geometry and 
fill static parameters. 


3.6.1 New Target Creation (“Target” sub-panel) 


This panel allows you to add, import and remove targets, and specify their geometry. 

Wp Fusion Studio Computer Vision Ee 
File View Mode ? 

ideo Manager (Camera Calibration ‘Video Preprocessing Tracking Manager (Output Manager 


Target 


Name Inst Geate 
Import... 


Remove 


Max Simultaneous Targets 


Video Rate: 29.84 Fps 


Figure 10 — “Scenario Manager” panel 
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In this step, you must know the type of object to follow in the video stream. The section “targets” (group box) 
contains targets management commands (see Figure 10): 
e “Create”: to add a new target to the list. Once the target is created, it is possible to change its name. 


e “Import”: to reuse a previously configured target. Notice that, for now, you can only import targets (and 
keyframes) created with the same camera properties. 


e “Remove”: to delete the currently selected target. All corresponding keyframes will be deleted as well. 


e “Max Simultaneous Targets”: to indicate how many targets can be recognized and 
tracked at the same time. 


Then the parameters included into the sub-panel “Target” must be filled: 


e “Object Type”: to declare the kind of tracker you want to use. Depending on the Mode (Lite or Classic), 
you can choose amongst “Plane”, “Face AutoInit”, “Object 3D” and “Plane Black Box” (Lite Mode 
only). 


e “Load model”: to import a wireframe (“.obj” or “.pts” files) that will be used as a virtual model of the 
real object. Available when using “Face AutoInit” or “Object 3D” object type. 


e “Plane Size”: to generate automatically a wireframe. Available when using “plane” object type. 
e “Scale”: to modify the scale of the object according to the X, Y and Z-axis. 
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3.6.2 Keyframes & Images Registration (“Keyframe” sub-panel) 


The goal of the keyframe registration process is to establish matches between the target shown in the video and 
a virtual model (represented by a green wireframe) corresponding to the geometry of the real object. This step 
can be achieve manually using a trackball or automatically using an image (.jog or .bmp) 


T D'Fusion Studio Computer Vision 


File View Mode ? 


Video Manager Camera Calibration (Vid j ac put Manager 


nultaneous Te 


Target keyframe Face AutoInit Pointing Detection ‘E-lgi 


Target "ype [lane 


lane Size w 10 


-Keyframe 


Target Keyframe Face Autolnit Pointing Detection |t% -lke) 


Create Import... Display 


Remove Imagefs}... Freeze Create Import... Display 


Remove Image(s)... Freeze 
Freeze delay 


Freeze delay 


-Keyframe Option 
Tracked Keypoints Number 
Y 0,00 
Y 0.00 
Reset Pose 


© Trackball Mode FPS 


Keyframe Info Trackball Speed 


Nb. Images per Group 
Pos; X% 06.00 Y 0.00 
Nb. Group Detections per Frame 
Rot: X -2.75 
Reset Pose Display Axis Nb, Max In Points per Frame 


© Trackball Mode FPS 
Trackball Speed 


Figure 11 — “Keyframe” sub-panel - left window 


(1) corresponds to the “Classic” tracking mode while right window (2) is displayed with the “Lite” 
mode. Notice that the red facets of the wireframe correspond to front faces and must be visible to the 


user. 
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In order to achieve this match, the user has to freeze the video stream and move correctly the virtual model with 
the mouse. The possible mouse movements in the main display window can help the user in this task: 

e A “left button drag’n drop” on this surface will rotate the model in a trackball fashion. 

e A “right button drag-n drop” will translate the model in a plane parallel to the screen plane. 


e A “middle button drag’n drop” or a use of the mouse wheel will translate the model in an axis 
perpendicular to the screen plane. 


The “keyframe” sub-panel contains the following elements: 
¢ “Up/down” buttons : to change the keyframe order. 


e “Freeze/Unfreeze video” button : to freezes the video in order to allow a fine positioning of the virtual 
model. 


e “Freeze Delay” : to provide a delay (in seconds) before freezing the video. 


e “Create” button : to creates a new keyframe based on the still video image and the current position and 
orientation of the virtual model. 


e “Import ...” button : to import keyframe from an older scenario. 

e “Image ...” button : to register a keyframe using an image (.4/mp or .jpg) available on the disk. 
e “Remove?” button : to removes the selected keyframe. 

e “Display” button : to display the selected keyframe instead of the video stream. 

e “Enable Modification”: to enable keyframe modification once created. 

e “Activate” check box : to Enable/Disable a keyframe when proceeding the tracking. 


e “AutoInit Keyframe” button : to activate the automatic initialization capabilities using the selected 
keyframe. 


e “Pointing Keyframe” button : to activate the pointing detection. 

e “Vertical Flip” button : to flip the selected keyframe vertically (works only with planar objects). 

e “Horizontal Flip” button : to flip the selected keyframe horizontally (works only with planar objects). 
e “Rotate” button : to create a new rotated keyframe (works only with planar objects). 


To move the wireframe in order to create a keyframe, you can use the buttons that belong to the “Keyframe 
Info” group box: 


e “X, Y, Z, RotX, RotY, RotZ” buttons: to specify directly the position and the orientation of the 
wireframe. 

e “Reset Pose” button: to reset the position and the orientation of the wireframe. 

e “Display Axis” button: to display the X, Y and Z-axis. 

e “Trackball Mode”: common trackball mode. 


e “Fps”: to activate the First Person Shooter Trackball mode in order to move the camera instead moving 
directly the wireframe. 


e “Trackball Speed”: to specify the speed of movements related with the mouse movements. 


Some parameters are specific to the “Lite” tracking mode: 


e “Tracked Keyframe Number” edit box: to set the number of features to be tracked on a keyframe (a 
value between 25..35 is recommended). 


e “Nb. Images per Group” edit box: to set the number of keyframes (images) stored in the same group 
(increasing this value will make the recognition faster but less robust). 


e “Generate Classifiers” button: generate statistical structures based on given keyframes to allow 
automatic initialization and recognition. 


e “Remove Classifiers”: to clean up previously generated classifiers (in case images have changed). 


e “Nb. Max Interest Points per Frame”: to specify the number of interest points to be used for the 
detection step. 
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3.6.3 Face with Automatic Initialization (“Face AutoInit” sub-panel) 


The “Face AutoInit” sub-panel can be used in order to adjust parameters corresponding to “FACE_AUTOINIT” 
objects: 


e “File Indice Point ...”: to select the file that must contain the correspondences between 2D human 
feature points and 3D points on the selected wireframe. 

e “Scale Model” : to apply a scale with the automatic initialization. 

e “Image Rotation”: to apply a rotation to computation if the camera is rolled around to its view axis. 


e “Initialization delay”: to add a delay in order to ensure that the user has enough time to put his head 
in a correct position in front of the camera. 


e “Face Classifiers”: to specify a head detection classifier that will be used by OpenCV face detection 
algorithm. 


¢ “Vertical Position” : to apply a vertical offset on the 3D pose of the detected face. 


Max Simultaneous Targets 1 
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Figure 12 — “Face AutoInit” sub-panel - These parameters are available with face tracking objects 


only. 
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In the “Lite” Mode, there are some more options that you can modify according to the scenario you want to play. 


“Nb. Max Interest Point Per Frame”: to specify the maximum number of interest point per frame. 


“Face Search Mode”: to select if you want to search a face at different scale by rescaling the image 
and applying a classifier which is supposed to find face at a fixed size in pixel or by rescaling the classifier 
to change the face size is should find without modifying the image. 


“Face Size Min”: to specify the minimum face size expected during the face detection step. 
“Face Size Max”: to specify the maximum face size expected during the face detection step. 
“ROI Edge Width”: to set the width of the left and right edges where the face isn’t searched (pixel). 


“ROI Edge Height”: to set the height of the upper and lower edges where the face isn’t searched 
(pixel). 

“Pyramidal face Search” : to specify that the search of the face at different scale is pyramidal : there 
is a factor of “Detection Scale Step” between to step of the scale search 


“Detection scale Step” : to set the scale factor between two scale step when the “Pyramidal face 
Search” is used 


“Linear face Search”: to specify that the search of the face at different scale is linear : there is a 
difference of “Detection Size Step” pixels between to step of the scale search 


“Detection Size Step”: to set the difference (pixels) between two scale step when the “Linear face Search” 


is used 
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Figure 13 — “Target” panel when target type is “Face AutoInit” in “Lite” tracking. 
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3.6.4 Pointing Detection Capabilities (“Pointing Detection” sub-panel) 


When using the pointing detection engine, you have to specify areas on the top of the keyframe to declare a set 
of “active areas” (See the Figure 14). 
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Figure 14 — “Pointing Detection” sub-panel. An area has been created. 


This process can be achieved using the “Pointing Detection” panel’s commands: 
e “Optimized Reprojection”: to use optimization for occultation computation (this box should be 
checked). 
e “Mouse functions”: to help the user to design new active areas directly on the rendered window. 
e “Zone” panel: to manage created areas. 
e “Delete Zone” button: to remove currently selected area. 
e “Priority” list: to specify the priority order when multiple areas are occulted at the same time. 
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3.6.5 Design Exclusion Areas (“Exclusion Areas” sub-panel) 


The user can specify polygonal areas on the top of the keyframe to declare a set of “non active areas” (see the 
Figure 15). 
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Figure 15 — “Exclusion Areas” sub-panel 


This process can be achieved using the “Exclusion Areas” panel’s commands: 


e “keyframe(s)”: to select a keyframe created for the current object. 

e “Area(s)”: to select an existing area. 

e “Validate”: to validate the current area that is drawn on the rendering window. 

e “Delete”: to delete an existing area. 

e “Use Exclusions For Tracking”: to use exclusion areas during the tracking step. 
e “Mouse left click”: to design an exclusion area on top of the rendering window. 
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3.7 PARAMETERS CONFIGURATION (“TRACKING MANAGER” PANEL) 


This panel can be used to test the scenario previously created and to modify parameters in order to improve the 
robustness and the performances of the tracking application. 


The “Tracking Manager” panel provides a list of created object. It’s possible to activate or deactivate them 
using the check box that appears on the left. 


Main options and parameters are the following; 


e The list of objects that were created. A check box is associated with the object name to disable it simply. 


“Max Simultaneous Targets”: to indicate how many target should be tracked at the same time using 
the video stream. 


“Use Exclusion Image”: to use an exclusion image to speed up the recognition step when trying to 
recognize and to track multiple objects at the same time. This option is not available with the “Lite” 
tracking since it is set automatically according to the user’s needs. 
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3.7.1 Target related parameters (“Target Tracking” sub-panel) 
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The “Target Tracking” sub-panel functions: 
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Figure 16 — “Tracking Manager” panel and “Target Tracking” sub-panel 


¢ ™Display Wireframe On Init” button: Displays/hides the wireframe. 
e “Display Keyframe” button: Displays/hides a keyframe. 
e¢ “Display Coordinate System” button: Displays/hides the object’s coordinate system. 
e “Face Cull Angle”: to remove, in the tracking process, facets that are aligned with the camera viewing 


direction. 
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e “Reprojection”: to specify the keyframe reprojection mode. When tracking 3D objects, this value is 
forced to “Local Patch”. 


e “Reprojection Threshold”: this threshold indicates when special reprojection modes must be applied. 
This parameter is available only with 2D objects and with “Reprojection” set to “Integral” or “Integral 
Interest Points”. 


e “Front/Back Clipping Plane”: to specify the front and back clipping plane to display the wireframe. 
These parameters are only active when activating the “Use GPU” option. 
e “Convergence Iteration”: to specify sub-image convergence frame numbers. 


e “Scanning Mode”: to specify the scanning mode amongst “Full Fixed” (the sub-image window is fixed 
all the time), “Init Fixed” (the sub-image window is fixed only during initialization), “Full Scan” (the 
sub-image moves during initialization and during the tracking). 
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3.7.2 Interest Points related parameters (“Interest Points” sub-panel) 
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Figure 17 — “Interest Points” sub-panel 


TOTAL «526, Av. du Général Charles de Gaulle Fusion CV Reference Manualdoc 32. 96 
IMMERSION ® +33 (0) 146 25 97 42 


& www.t-immersion.com 


D’Fusion Computer Vision 
TOTAL IMMERSION « ” P 11/12/12 


User Manual 


This panel contains parameters that can be changed in order to reduce or to increase the number of feature 
points that are detected on the current frame and on the tracked objects : 


¢ “Display Keyframe/Previous Interest Points”: to display feature points on the top of the video 
stream. This check box should be checked when the user wants to change the following parameters. 
Displayed colours (when tracking) : 
¢ Green indicates a good match 
e Red indicates a non match 


e Blue indicates a detected good match that finally turned into a bad match (due to local similarity, 
noise or errors). These blue points are usually bad for the tracking quality. 


e “Max Number”: to specify the maximum number of feature points that can be retrieved on the current 
image. 

e “Quality”: to indicate the quality of feature points that should be retrieve on the current frame. A value 
close to 1 indicates that sharp edges will only be detected, while a value close to 10 indicates that round 
edges will be identified as well. 

e “Alpha”: to specify if feature points should be contrasted on a neighbourhood. A high value (greater 
than 0.1) indicates that only strong feature points are recovered. 

e “Init Max Number”: to indicate how many feature points should be used for non automatic initialization 
process. 

e “Window Size, Derive Size”: to specify the neighbourhood taken into account for the feature point 
detection process. 

e “Number of Regions”: to spread detected feature points uniformly on the current image. A value of 10 
will create 100 regions and spread detected points among the image. 
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Figure 18 — “Correlation” sub-panel 
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This panel contains parameters that can be changed in order to increase the number of good matches between 
identified feature points: 


¢ “Display correlation”: to display correlation information in a debug window. 


e “Max Disparity U & V”: to increase the robustness against large displacements. Notice that increasing 


these values will introduce potential match errors. 
e “Window Size”: to specify the number of pixels to be proceeded for the correlation calculation. 


e “Threshold”: to indicate the correlation threshold. A large value will increase the number of potential 
matches but will also introduce errors. 
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3.7.4 Keyframes related parameters (“Keyframe(s)” sub-panel) 
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Figure 19 — “Keyframe(s)” sub-panel 
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identified feature points: 


“Init Match Threshold”: threshold to indicate the number of feature points that have to be matched in 
order to proceed the non automatic initialization. 


“Correlation Threshold”: threshold to indicate the number of matches with the keyframe required to 
stay in the tracking step. 


“Failure Threshold P”: threshold to indicates the number of correct correlations with the previous 
frame required to stay into the tracking state 


“Inliers Threshold (Recursive)”: threshold to indicates the number of correct correlations (inliers) 
with the currently selected keyframe required to decide to proceed the recursive only tracking mode. 


“Keyframes Switch Bias”: threshold to avoid fast switching between created keyframes. 


Ransac Initialization section allows to modify the parameters for the keyframe initialization mode. 


“Iteration”: to specify the number of iteration to match the keyframes. 
“Inliers Threshold”: to indicate the maximum distance to validate a match. 


Online Keyframes section helps to setup the online keyframe creation functionality. 


“Use Online Keyframe”: to enable online keyframe creation. 


“Mode” : to indicate the way new online keyframes are created. The “Inliers” mode is dedicated to 
create online keyframes when too few feature points are match during the tracking, while the “Rotation” 
mode aims to create keyframes only when the object moves too much in front of the camera. 


“Nb Inliers to Create KF” : threshold to indicate when the tracking process should create the new 
keyframes. A large value implies a large number of online keyframes. 


“Angle, Distance” : threshold to indicate when the tracking process should create new keyframes. 
These edit boxes is available only with the “Rotation” mode. 


“Max Online Keyframe” : to specify how many keyframe can be created during the tracking process. 
“Auto Delete” : to delete online keyframes when the tracking is lost. 
“Prefer Offline Info” : to increase offline keyframes influence when possible. 


“Difference off/on” : threshold to indicate the difference between feature points number on online 
keyframes in order to decide to use them instead off using offline keyframes. This parameter is only 
available when the “Prefer Offline Info” check box is checked. 


“Create”: to create an online keyframe while tracking. This keyframe will be store as an offline keyframe 
in the corresponding object. 
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3.7.5 Pose Estimation related parameters (“Pose Estimation” sub-panel) 
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Figure 20 — “Pose Estimation” sub-panel 
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The “Pose Estimation” sub-panel is dedicated to choose a process in order to estimate the object position and 
orientation in front of the camera. Using these functionalities, the user will have to deal between performances 
and robustness: 


e “Tracking Method”: to choose between “Performances” or “Robustness” for this pose estimation 
process. 

e “Use Weighted iteration”: to indicate if the process should be very robust. This check box should be 
check when high quality of the tracking is mandatory. 

e “Use Weights on Correlation”: to indicate if the process of matching uses more weight on best 
matches. 

e “Use Weights on Borders”: to indicate if the process of matching uses more weight on matches that 
are close to object borders. 

e “Use Ransac For KF Pose Estimation”: to indicate if the RANSAC algorithm is used to remove errors 
using the keyframe pose estimation. 

e “Use Ransac For PKF Pose Estimation”: to indicate if the RANSAC algorithm is used to remove errors 
using the previous keyframe pose estimation. 

e “Recursive Tracking Frame”: to indicate how many frame should be used to apply the recursive KLT 
tracking. This parameter is taken into account only if the user checked the “Use KLT Recursive Tracking” 
check box. 

e “Use KLT Recursive Tracking”: to indicate if the recursive KLT tracking is used. Using this special 
tracking mode will ensure the tracking robustness with web cameras. 
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3.7.6 Automatic Initialization related parameters (“AutoInit” sub-panel) 


Instances 
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-Sub Image 


Convergence Iteration 


Scanning Mode 


Figure 21 — “Auto Init” sub-panel 


The “Auto Init” sub-panel is dedicated to the automatic initialization engine: 


e “Search Mode”: to change the priority order of the recognition service. Possible choice are : 


e “Declare Order”: keyframes are processed in the given order and the process stops when a match 
is found. 


e “Previous then declare order”: keyframes are processed in the given order but starting with the 
previously found one. Then the process stops when a match is found. 


e “One detection by trame”: a unique keyframe is processed each frame, 
e “N detections by trame”: N keyframes are processed each frame. 


e “Nb. of detections by Trame”: to specify the number of recognition to be proceeded at each trame. 
This edit box is available only of the “Search Mode” is set to “N detections by trame”. 
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“Use Cropped Win”: process sub divided images to help recognition when the object is too small in the 
current image. This option is not available with the “Lite” tracking. 
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3.7.7 Pointing Detection related parameters (“Pointing Detection” sub-panel) 
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Figure 22 — “Pointing Detection” sub-panel 
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Using the “Pointing Detection” sub-panel allows specifying the way pointing detection areas will be detected: 
e “Buffer Size”: Size of the buffer to retain successive pointing detection results in order to be able to 
filter noise on the pointed areas. 


e “Nb of Same Id Min”: Minimum number of same area index required in the buffer in order to validate 
that an area is occulted. 


e “Nb of Same Id Consecutive”: Minimum number of consecutive area index in the buffer in order to 
validate that an area is occulted. 


e “Nb Negative”: Maximum number of false detection in the buffer to decide that there is no occultation. 
e “Keyframes” list: list of keyframes that have the pointing detection process activated 

e “Use Debug Mode”: to display a window that contains pointing detection areas. 

e “Use GPU”: to use the GPU for the pointing detection process. 

e “Move Threshold”: threshold (in pixels) to disable the pointing detection on fast movements. 

e “Zone(s)” list: list of active area that are available for pointing detection. 

e “Win Size Loc”: Correlation window size for point localisation in current frame(in pixel). 


e “Threshold Loc”: Correlation threshold used for point loclisation. If correlation result doenst reach this 
value, the point is not loclized ans hidden. 


e “Win Size Dec”: Correlation window size used to decided if a localized point is hidden. The value should 
be larger than corrWinSizeLoc 


e “Threshold Dec”: Correlation threshold used for decision. If correlation result doenst reach this value, 
the localized point is hidden. 


e “Detect Threshold”: Ratio between localized point number and tolal point number. If the ratio is 
smaller than this value, occlusion is detected. 


e “Search Area Size”: Search area size for point localisation (expressed in pixels). 


e “Ip Dist”: Minimum distance (in pixel) between 2 interest points in the active area. A smaller value will 
create more feature points on the area. 


e “Ip Quality”: Interest point detection quality parameter. 
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3.7.8 Move Detector related parameters (“Move Detector” sub-panel) 
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Figure 23 — “Move Detector” sub-panel 


The “Move Detector” can be used to stabilize the object position and orientation when both the camera and the 
object are fixed: 

e “Use Move Detector”: to activate or deactivate the move detector process. 

e “Nb Frame Init”: to indicate the number of frame required before fixing the object pose. 


e “Fixed Point Radius”: to indicate the neighboring distance to search for stability. A lower value will 
make move detection more difficult while a larger value will introduce some steps artifacts in the pose 
estimation process. 


e “Min Nb Fixed Points”: Number of fixed points required before fixing the pose of the object. 


e “Area Percent Required”: Percent area (of the convex hull) of fixed features required relative to the 
area (of convex hull) of stable points. 


e “Max Features”: Maximum of features point that can be used for the move detection analysis. 
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3.7.9 Process related parameters (“Process” sub-panel) 
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Figure 24 — “Process” sub-panel 


In this “Process” sub-panel, it is possible to change the priority of the tracking thread: 


e “CPU”: to change the CPU used for the tracking. -1 will let windows™ decide. 
e “CPU Priority”: to fix the priority of the tracking thread. 


3.7.10 Results related parameters (“Results” sub-panel) 


This sub-panel presents results of detected/tracked objects. First are displayed Position and Rotation information 
of the active object. Then for each object, a statement is displayed at the bottom of the sub-panel. 
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3.7.11 Output configuration (“Output Manager” panel) 


This panel is dedicated to help users to use the MarkerLess Tracking resulting information. 
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Figure 25 — “Output Manager” panel 
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As described in the figure, it is possible to modify the data that are sent to the rendering application: 
e “No Network Output” check box: this mode deactivate the UDP sender and is usually used with 
D’Fusion Augmented Reality plug-in. 


e “Send Synchronous” check box: this mode is dedicated to send UDP data in a synchronous way 
according to the tracking process. 


e “Send Asynchronous” this mode is dedicated to send UDP data in an asynchronous way according to a 
specified rate (see the buttons “frequency rate” and “phase” bellow). 


e “Local Port” edit box: specify to local port to receive UDP packets (usually sent by D’Fusion Augmented 
Reality). 


e “Ip Address” list: specify up to 8 hosts to send UDP packets. 


e “Send Mode” list: choose between object coordinates expressed according to the camera referential or 
camera coordinates expressed according to object referential. 


e “Send Format” list: UDP output format to be choose between D’Fusion Augmented Reality format 
(usable with D’Fusion Augmented Reality and Brainstorm®’s plugin), and D’FusionDragon® Format 
(usable with VizRT®’s plugin). Notice that there exists a “Standard” format that sends directly the 
tracking results without any referential conversion. 


e “Orientation Send Filter” matrix: describe final operation on homogeneous rotation vector [RotX, RotY, 
RotZ, 1]. 


e “Translation Send Filter” matrix: describe final operation on homogeneous translation vector [X, Y, Z, 
1]. 


3.8 USER LEVEL 


When using the “Lite” tracking, the user level can be changed using the menu “User Leve/ -> Basic, or “User 
Level -> Advanced. In case you select the Basic user level mode, the D’Fusion Studio CV Pro is changed to a 
simpler version. It should help beginner to design very standard scenarios. 


In this basic user level, you will have only access to mandatory options such as : 


e Asimplified list of panels (Video Manager, Scenario Manager, and Tracking Manager). 
e Simplified functionalities since only very useful panels are available. 
Using the basic user level, you will not be able to: 


e Specify the camera calibration 
e Specify the processing data 
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4. HOW-TO AND USE CASES 


4.1 CONFIGURATION STEPS 


In this section a reminder of each step of the configuration process is provided. According to your requirements 
you may be able to adapt the following scenarios to your tracking. All these steps (except for the 3D modeling) 
can be done using D’Fusion Studio CV Pro. 


1. Video Signal Configuration 
The very first step to achieve is to setup the video capture and preprocessing parameters. According to 
your camera and computer specifications, you will have to modify a few parameters. 

2. Camera Calibration 
The Camera Calibration tool is available through the “Camera Calibration” panel. It will allow you to adjust 
the physical characteristics of your camera. Once the calibration is done, you will have to report the 
calibration values through the “Video Processing” panel. 

3. Virtual Model Creation 
A virtual model (a 3D mesh) can be created using standard DCC tools. The resulting 3D format must be 
saved as .obj (Wavefront OBJ object). A link to this 3D file must be created through the “Scenario 
Manager” panel. Another solution is to use planar targets and specify directly the target width and the 
target height. 

4. “Lite” vs “Classic” Mode 
Depending on the targeted platform, you will have to choose between the “Classica” tracker and the “Lite” 
tracker using the “Mode -> Classic” or “Mode -> Lite” menu. The Lite tracker has been designed for Mobile 
and Flash applications, while the Classic tracker is the one used for standard applications. 

5. Object (Target) Creation 
Create new objects (targets). A target is attached to a specific geometry. In your scenario, it’s possible to 
add multiple objects and to track them at the same time in the video stream. To do that, you can use the 
“instance” functionality in order to duplicate objects. 

6. Keyframe Creation 
Create and register new keyframes that correspond to the real object appearance. 

7. Set up the tracking parameters 
Adjust the tracking parameters through “Tracking Manager” panel. 

8. Set up the output 
If you wish to display MLT resulting information in your application, you will have to set up output 
information using the “Output Manager” panel. 

9. Launch the application 
When launching the D’Fusion CV Player, a new window should appear with the captured video signal 
displayed. Then when the real object is moved in front of the camera, a wireframe grid should appear on 
the top of that object. This means that you entered the tracking step and everything works fine. 
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4.2 CAMERA CALIBRATION 


You can use the “Camera Calibration” panel available in D’Fusion Studio CV Pro to calibrate your camera. The 
following steps will guide you through the whole process. 


e Chessboard pattern selection: print a pattern using the “Print Pattern” button. Select a mire from 
the list. 


a 
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Max Nb Frame To Process 
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Use Estimation 


Zero Eangent Distortion 
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Acquisition Sech * 
Acquisition Sech ¥ 
Launch Pause Process 


Stop 
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e Chessboard characteristics configuration: the chessboard pattern selected and printed has its 


own characteristics. As a consequence, the edit boxes “Width”, “Height”, “Nb Square On Width”, 


“Nb Square On Height” must be filled according to the size and the number of black & white 
rectangles on the chessboard. 
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e Calibration process: to process the calibration, first specify the number of measurements required 
and error limits. Then click on the “Launch” button. 
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¢ Validation: check the camera parameters that result from the calibration process using the “camera 
sub-panel”. Then apply the calibration calculations to the MarkerLess Tracking by clicking on the “Set 
Camera To Scenario” button. Do not forget to save your camera calibration (using the “Save Camera” 
button) in order to be able to use it later in other projects. 
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Calibration using known Field of View: 


Internal camera information from D’Fusion Studio Computer Vision is given using classical OPENCV format using 
focal values (fx, fy) in pixels and the optical center position in pixel (cx, cy). But the user can convert these values 
into FovX and FovyY in degree you should use the following formula : 


FovX (degrees) = 2*htan(w/(2*Fx))*180/PI 
FovY (degrees) = 2*htan(h/(2*Fy))*180/PI 
And its inverse to obtain the focal values in pixels according to the Field of View: 
Fx (pixels) = w/(2*tan(FovX/2)) 
Fy (pixels) = h/(2*tan(FovY/2)) 
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4.3 PROCESSED IMAGE SPECIFICATION 


Once the camera is correctly calibrated, the user needs to decide the quality of the images (extracted from the 
video capture) that will be processed. This specification will be achieved through the “Video Processing” panel. 
« Video Processing » panel 


Import Calib,.. 


Camera 

Optical center (x, y) 320 240 
Focal (x, y) 1000 1000 
Fay (x, ¥} 35.4893 26,9915 
Max (x, ¥) 640 480 
Distortion 

Radial (a1, a2) 


Tangential (k1, k2) 


Activate distortion 


Processing size (x, ¥) 320 


l Skip odd or even Frame 


Sub Image 
Y Use SubImage 


Position (x, ¥) 


1. Check that your camera parameters correspond to the camera you are using. As an intuition, fx & fy should 
have approximately the same value. The value of fx & fy should be close to “MaxX” and “MaxyY” in case you 
are using a standard camera. In case you are using a narrow angle camera, these values should be less than 
“MaxX” and “MaxY”; and if you are using a wide angle camera, these values should be greater than 
“MaxX” and “MaxyY”. 


2. Indicate if you want to use image de-distortion using the “Activate Distortion” check box. Activating the 
distortion management is relevant if the camera has lenses that generate distortion effects on the processed 
image. This is often the case with wide angle cameras. Notice that de-distorting images is computationally 
expensive and the user should ensure that performances are good enough. 


3. Select the Processing Size of the processed image. Since it is not always possible to treat every single pixels 
acquired from the image sensor, it can be relevant to sub-sample the input image in order to save 
computational costs. As an example, if you use a 640x480 video signal input, you may process a 320x240 or 
a 640x480 image, depending on the targeted computer characteristics. When using HD images (eg. 
1920x1080 in interlaced), the user should select a 480x270 or a 960x540 tracking resolution. Notice that this 
is not mandatory to keep the aspect ratio given by the input image (or the image sensor). 


4. In case the input image is interlaced, it is possible to skip odd or even frame in order to stabilize the 
performances (using the “Skip Odd or Even Frame” button). 
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4.4 SWITCHING BETWEEN LITE & CLASSIC TRACKING MODE 


Depending on the targeted platform, you will have to choose between the Classic tracker (standard platforms 
such as PC and MAC) and the Lite tracker (with Adobe flash and mobiles). 


Below you will find a screenshot that explains how to switch between the 2 modes. 


 D'Fusion Studio Computer Vision 


File View Mode ? 


Y Classic 
Yideo Manager ¢ 


Lite 


The Lite tracker has access to fewer functionalities and some parameters become inaccessible. As a consequence, 
these will be disabled from the D’Fusion Studio CV Pro interface. 


In case the Lite tracking mode is mandatory for your application, you may need to modify a few specific 
parameters using the authoring tool: 


1. The number of feature points to be used during the tracking. The default value is 35, which ensures 
robustness without too much computation. You can carefully decrease this value to improve performance. 


Create 
Import.., 


Remove 


Max Simultaneous Targets 


Target 


Plane 


Nb. Max Interest Points per Frame 300 


| : 
oe ai B 
Create Import... Display 
Remove Image(s)... Freeze 
Freeze delay 
Tracked Keypoints Number 35 
3 00 Y 0.00 Zz 0.00 
eae |S, ¥ 0.00 ~ 0.00 
Reset Pose Display Axis 
Trackball Mode FPS 
. Imac al 
1, Gra Jetections per Frame 
Generate Classifiers Remove Classifiers 
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2. Indicate if you wish to use a RANSAC pose estimator in order to reduce parallax inversions. This 
parameter should be enabled only when the calibration is perfectly set and when the flyer is rigid (it 
cannot be deformed). These parameters are available in the panel “Tracking Manager”/sub-panel “Pose 
Estimation”. 


Instances 


Max Simultaneous Targets 


~ Use Exclusion Image 


yr amet Estimation ayito Init 


Minimization 
Use Weighted Iteration 
Ransac 
~ Use Ransac for Pose Estimation 
Iteration 


Inlier Reprojection Threshold 


3. In case you have many different objects to be recognized, you can group these objects in order to make 
the recognition step faster (with a loss of robustness). To do so, you will have to set the “Nb. Images per 
Group” parameter (see picture hereunder). Notice that when you are saving the scenario, a “.bin” file is 
generated for each group. 
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Target 


Name Inst Create 


Import... 


Remove 


Create Import... Display 


Remove Image(s),.. Freeze 


Freeze delay 


Tracked Keypoints Number 
¥ 0.00 


Y 0.00 


) Detections per Frame 4 
Generate Classifiers Remove Classifiers 


Nb, Max Interest Points per Frame 300 


When using the Lite recognition functionality, you will have to specify the size of the classifiers 
constructed per object. You can choose a size between 125KB to 4MB. A 125KB classifier will be less 
robust than a 2 or 4MB classifier. As a consequence, if you have only few objects to recognize (for 
example less than 20), you can easily reduce the size of the classifiers to 125 or 250KB. In case you have 
more than 20 objects or in case your objects look similar, you will have to create 1MB (or more) 
Classifiers. The following pop-up window will appear each time the “Generate classifiers” button is hit. 


VW Classifiers properties 


Quality 


OK = Cancel 
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4.5 USING THE INITIALIZATION WITH KEYFRAMES 


This section describes how to use initialization with keyframes once your camera has been calibrated. The 
initialization with keyframes mode must be used to track 3D shapes into the video stream, but can be also 
interesting when tracking planar objects (see Figure bellow for an example of planar and 3D targets). 


Figure 26 — Example of planar and 3D targets 
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(2D) PLANAR TARGETS - steps: 


Inst Create | 5 


Max Simultaneous 


Target keyframe Face Autolnit Pointing Detection » -lhwi 


Target Type Plane 


Plene Sive w 10 


{ Create Import... 


J Remove 


Freeze delay 


Keyframe Option 


v 


Current Video Frame 


Pointing Keyframe 
Yertical Flip Horizontal Flip 
Rotate 


-Keyframe Info 


6.00 


Reset Pose Display Axis 


Trackball Mode 


Trackball Speed 


1. Create a new object (button “Create”) and rename it using the GUI. 
2. In the sub-panel “Object”, select the object type: “plane”. 


11/12/12 


Object Keyframe Face Autoinit | Pointing Dete » ic 


3. Specify the object real size using your unit system (centimeters, meters, inches ...) in the “Plane Size” edit 


boxes. 


4. In the sub-panel “Keyframe”, capture the current video frame using the “Freeze” button with the object in 


the front of the camera. 
Move the wireframe using the trackball, to fit with the real object. 


6. Push the “Create” button to register your object using this current keyframe. 


You can create multiple keyframes for the same object to optimize initialization. In Classic mode, all these 
keyframes must be created from the same flyer (i.e. each keyframe represents the same flyer in a different 
view or position). In Lite mode, you can create multiple keyframes representing different images for the same 


object (each keyframe then initializes a different flyer). 


7. Test the tracking using the “Tracking Manager” main panel. 
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3D TARGETS -— steps (works for both Lite/Classic modes): 


Inst Create 
: Import... 


Remove 
) 
] 
} 


3 ject Keyhrame Face AutoInit | Pointing Dete» ir 


lax Si Ss Targets 2 fs = : 
Max Simultaneous Targets S) ObjectTotrack_keyframe_1 


\ NN ee ee 
Target KeyframeiFace Autolnit Pointing Detection |e» -lkgi 
Target Type 3D Object 


Load Model 


Create Import... Display 


Freeze 


Current Video Frame Freeze delay 


Keyframe Option 
4 
Pointing Keyframe 
Vertical Flip Horizontal Flip 
Rotate 
Keyframe Info 
Pos; ® 0.00 £ 0,00 


Rot: * 0.00 ¥ -1.57 Z 0.00 


Reset Pose Display Axis 


Trackball Mode 
Trackball Speed 


Create a new object (button “Create”) and rename it using the interface. 

In the sub-panel “Object”, select the object type : “object 3D”. 

Create the 3D model of your object and export it in an .obj format. 

Specify this previously created 3D model file using “the Load Model ...” button. 


OP oe ts I. 


In the sub-panel “Keyframe”, capture the current video frame using the “Freeze” button with the object in 
the front of the camera. 


6. Move the wireframe using the trackball, to fit with the real object. 
7. Push the “Create” button to register your object using this current keyframe. 
8. Test the tracking using the “Tracking Manager” main panel. 


Caution: When creating the keyframe using the trackball, the user must pay attention to the wireframe 
orientation. The risk is to orient the 3D grid with back triangles in the front of the camera. In this case, the 
D’Fusion CV Player will not be able to retrieve the target into the captured image. To ensure the wireframe has 
the right orientation, take care that the red facets of the wireframe are visible. 
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4.6 USING THE AUTOMATIC INITIALIZATION MODE 


This section describes how to use Automatic Initialization with planar targets. When Automatic Initialization is 
enabled, the initialization step becomes easier since the user doesn’t have any constraint regarding the initial 
pose of the real object in front of the camera. 


Once the camera is correctly calibrated, the following preparation steps are required to allow a fully automatic 
initialization. 1) You will have two ways to register objects: using a keyframe or using an image from the disk. 2) 
Once images or keyframes are registered, classifiers corresponding to the data will be created and stored on disk. 
If you wish to change registered images, you will have to recompute the corresponding classifiers. 

— = 


-~ 


20 eR A Oni SATE 


Figure 27 —The given image (a .jpg or .bmp file) and the corresponding keyframe 
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CREATE AN AUTOMATIC INITIALIZATION OBJECT USING A KEYFRAME (using a screenshot of the real 


object): 
eTarget —_ _—_™@]@ —_ —J 
Name Inst Create 
Import... 


Remove 


7) 


Target keyframe |Face Autolnit Pointing Detection |E» -l\wi 


Target Type = Plane » 


Plane Size w 10 h io 


1. Create an object and a planar keyframe as specified in section 4.5. 
2. Select the “Plane” object type in the “Object” sub-panel. 


3. Check the “AutoInit Keyframe” button in the “Keyframe” sub-panel, and wait until the automatic 
initialization generation process is finished (see the progress bar at the bottom of the screen). In case you 
are using the Lite mode, you will also have to set the size of the classifier (the smaller the size is, the less 
robust it is for the recognition step). 


4. Test the tracking using the “Tracking Manager” main panel. 
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CREATE AN AUTOMATIC INITIALIZATION OBJECT USIN AN IMAGE: 


Target AAA 
Name Inst Create | 
Import... 


Remove 


Max Simultaneous Targets 


O& Keyframe!Face Autolnit Pointing Detection 


Target Type = Plane 


Plane Size w 10 P 10 


1. Create an object and rename it. 


D’Fusion Computer Vision 


11/12/12 


2. Select the “Plane” (or “Plane AutoInit’ within the Lite mode) object type in the “Object” sub-panel. 


Specify the object real size using your unit system (centimeters, meters, inches ...). Notice that this 
parameters should be as accurate as possible and that the aspect ratio between the loaded image and the 


printed image must be the same. 


4. Load one or a plurality of .Smpor a .jog from the disk using the “Image...” button in the “Object” sub-panel. 
Notice that this image should have been resized according to the processed size and according to the 
approximate size in pixels of the object in front of the camera (this must be done outside the D’Fusion Studio 
Tool). For one Lite Plane Target, all the keyframe images must have the same aspect ratio, corresponding to 


the target aspect ratio, otherwise it won't function. 
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The “AutoInit Keyframe” button in the “Keyframe” sub-panel should be automatically checked, then you 
will have to wait few minutes before computation ends. In case you are using the Lite mode, you will also 
have to set the size of the classifier (the smaller the size is, the less robust it is for the recognition step). 


6. Test the tracking using the “Tracking Manager” main panel. 
Caution: When using images to register new object, the creation step is fast, but you need to ensure that the 


given image “looks like” the real image of the target in the video stream. Otherwise, the initialization step may 
have some problem to succeed. As a consequence, the user should verify that the tracking works correctly. 
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4.7 USING THE MULTI-TARGET TRACKING MODE 


This section describes how to construct a multi-target scenario. Notice that in this case, each target can have 
a different geometry and a corresponding set of keyframes. Using the GUI, it’s simple to create multiple objects, 
to assign a specific geometry and corresponding keyframes. 


Two options are available in the scenario creation management to create a multi-target scenario. The first one 
should be used when objects have different geometries (for example: a face and a card, or two cards with 
different aspect ratios). The second one is easier since we will have the same geometry and applied the same 
parameters to every objects. In this case the user should use the “instances” number to duplicate an 
object. 


OPTION 1: MULTIPLE OBJECTS TRACKING WITH DIFFERENT GEOMETRIES 
1. Create new objects and register corresponding keyframes as explained in previous sections 4.5 & 4.6 
2. Test the tracking using the “Tracking Manager” main panel. 
3. Select the objects you want to track using the main list check boxes. 
4 


Select the number of objects you want to track at the same time using the “Total Target To Track” edit 
box. 


Caution: When using multiple target capabilities, the object name become important since it identify the output 
data packets to be sent to the renderer software. 
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« Tracking Manager » Panel 


w Use Exclusion Image 


Display 
Display 


--Rerender 
Face Cull Angle 
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Reprojection 
Reprojection Threshold 
Front Clipping Plane 


Back Clipping Plane 


--Sub Image 


Scanning Mode 


Local Patch 


Instances 
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OPTION 2: MULTIPLE OBJECTS TRACKING WITH SAME GEOMETRY 
In this case, the creation process for a new object is much easier. 
1. Create a new object and register corresponding keyframes as explained in previous sections 4.5 & 4.6 
2. Duplicate the object by modifying the instance number. 
3. Test the tracking using the “Tracking Manager” main panel. 
4 


Select the number of objects you want to track at the same time using the “Total Target To Track” edit 
box. 


8 8 —<$<$—$—————————————————— 
Name Inst Create 
Import... 


Remove 


: 


Target |KEyFrame Face Autolnit Pointing Detection /E» aig) 


Create Import,,. 
Remove Imagets}... 


~ Freeze delay 


-Keyframe Option 
Ww 
~ Autolnit keyframe 


Pointing Keyframe 


Vertical Flip Horizontal Flip 


Rotate 


ckKeyframe Info 
Pos; % 0.00 0.00 
Rot: ® 3,14 0.00 
Reset Pose Display Axis 


Trackball Mode 


Trackball Speed 
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4.8 USING THE RECOGNITION CAPABILITIES 


The recognition module is dedicated to retrieve one specific registered target among a plurality. To design a valid 
scenario using the recognition capabilities, the user will have first to register planar objects using auto- 
initialization as described in the section 4.6. Up to 50 objects can be registered into the system using D’Fusion 
Studio CV Pro. 


-Target 
Name 


« Tracking Manager » Panel 


Name Instances 
lv 

ww Autolnit_ 

w Autalnit_| 


li 


Max Simultaneous Targets 


Use Exclusion Image 
5 


Sinks Correlation Keyframe{s) Pose Estimation | AUC INIC | irie- 
Create Import... Display Parameters 
Remove Image(s)... Freeze Search Mode Previous Then Declar Orc 
Freeze delay 0 Nb, of Detections by Frame 


Use Cropped Win 


--Keyframe Option 


v 
 Autolnit Keyframe 
Pointing Keyframe 


Vertical Flip Horizontal Flip 
Rotate 


eKeyframe Info 
Pos: X 0.00 ¥Y o.oo Z 20.00 
¥ 0.00 @ 0.00 
Display Axis 


© Trackball Mode 
Trackball Speed 


1. Register new objects and corresponding new keyframes. 
2. Test the recognition and the tracking using the “Tracking Manager” main panel. 


3. Adjust the priority order using the “Search Mode” list in the sub-panel “Auto Init”. If you use many 
object (more than 20), it’s possible to use the “N Detections By Frames” in order to reduce the number 
of recognition proceeded at each frame and to save performances. 


4. Use the exclusion image manager by clicking the “Use Exclusion Image” check box in case you are 
tracking multiple automatic initialization targets at the same time. 
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4.9 USING THE AUTO INITIALIZATION MODE 


This section describes how to use the face auto initialization capabilities. This tracking mode can recognize any 
face. To use the face auto initialization follow these steps: 


w 


4 
5. 
6 


7. 


Inst Create | 
Import... 


Remove 


) Object }Keyfrome Face Autolnit pointing Dete » i 


Max Simultaneous Targets 


Mode Auto Init 


File Indice Point 
Target Keyframe !Face Autolnit Pointing Detection f= -lhsi 
Scale Moc's| 


Target Type Face Autolnit Image Rotation 


Load Model’ visage.x.pts Initialization Uslay 5 

Face Classifier ‘ontalface co 
Right Eye 

Left Eye 


Nose 


Current Video Frame Mouth 


Create a new object and rename it 
In the “Object” sub-panel, select the “Face AutoInit” object type. 


A default wireframe corresponding to human faces is loaded. You can replace it by your own 3D model if 
required. 


In the “Face AutoInit” sub-panel, adjust the model scale (using the “Scale Model” text field). 
In the “Face AutoInit” sub-panel, adjust the image rotation (in case your camera is rotated). 
Adjust the delay before initialization using the “Initialization Delay” edit box. 


Launch the tracking using the “Tracking Manager” panel to check if the tracking works correctly. 


In case the tracking is not robust enough, the user should adjust the following “live” parameters from the 
“Tracking Manager” panel: 


8. In the “Target Tracking” sub-panel, modify the “Face Cull Angle” value to 85 in order to remove facets that 
are parallel to the camera viewing direction. 
9. In the “Target Tracking” sub-panel, ensure that “Pick Nearest Facet” is checked and that the 
“Reprojection” mode is set to “Local Patch”. 
10. In the “Interest Points” sub-panel, set the “Max Number” feature points to 500 or 600, and the “Quality” 
to 8 or 9. 
TOTAL ed ae eta al Charles de Gaulle DFusion CV - Reference Manual.doc 67 / 96 
IMMERSION @ +33 (0) 146 25 97 42 


& www.t-immersion.com 


D’Fusion Com r Vision 
TOTAL IMMERSION « ” <% puter Visio 


11/12/12 
User Manual 
« Tracking Manager » Panel « Tracking Manager » Panel 
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Display - Parameters 
Display Wireframe on Init 
Displ Pisin Display Keyfram est Points 
Display Coordinate Sy Dervay Srovinus Front tiroreec is 
Max Number 500 
-Rerender 
Quality 
Face Cull Angle 
Alpha 0,10 
~ Pick Near .: 
Reprojection Local Patch Init Max Number 300 
Reprojection Threshold 0,50 Window Size 
Front Clipping Plane 0,10 
Back Clipping Plane Number of Regicnis) 
- i he Image 
Convergence Iteration 
Scanning Mode 
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4.10 USING THE SUB-IMAGE MODE 


Notice that this functionality is available only with the Classic Tracking Mode. 


This section describes how to use the sub image mode. This tracking mode is useful to recognize and follow small 
objects into the video stream because the video stream is processed without been downscaled. This mode is also 
useful when using wide angle camera. 


« Video Processing » panel == a8 os | 4 « Tracking Manager » panel 


Import Calib,., —Parameters 


camera 


Target Tracking: | Trt, 
Optical center (x, y) 


Focal (x,y) Total Target To 


Fay (x, ¥} 


Max (x, y) 
| 
SSS SSeS ST 


Display 
Display WireFframe on Init 
Display Keyframe 


Display Coordinate System 


Distortion 


Radial (a1, a2) -Rerender 


Tangential (k1 Use GPU 


Activate distortion Face Cull Angle 


Preprocessing —£§—2-@- @@4————~ v Pick Nearest Facet 
/ Reprojection Local Patch 
Processing size (x, ¥ 320 | 


Skip add or even Frame Reprojection Thresho 


\ 


Front Clipping Plane 


Sub Image 
Back Clipping Plane 


—__——— 


| 
d y’ Use SubImage 
l Position (x 
\ 


--Sub Image 


Convergence Iteration 


Scanning Mode Full Scan 


Choose the sub-window pre-processing size in the “Video Processing” panel. 
Declare that you want to use the sub image mode by checking the “Use SubImage” check box. 


Move your sub-image area using the mouse in the rendering window or using the “X:” and “Y:” edit boxes. 


a er 


Create your objects and keyframes as described in previous sections and test the tracking using the 
“Tracking Manager” panel to check if the tracking works correctly. 


5. If you want to fix the sub-image scanning change the mode in the “Scanning Mode” list. 
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4.11 USING THE POINTING DETECTION MODE 


Notice that this functionality is available only with the Classical Tracking Mode. 


The pointing detection is a module provided to detect perturbations that appear over the tracked target. As a 
consequence, this functionality works only when the object is tracked in the video stream. Be also careful that this 
functionality only works with planar objects. 


To add the pointing detection functionality in your scenario, you need to execute the following steps using the 
“Scenario Manager” and “Tracking Manager” panels: 


Manager » Panel 


« Scenario Manager » Panel « Scenario Manager » Panel = = 2/6 == Pb» « Trackinc 
Target Target A — Thakances 


: vy voicing. obet 
Name Inst Create Name Inst Create : 
Pointing Object Import... ro Import... 


Remove Remove 


Max Simultaneous Targets ultaneous Targets 1 


SEE aa 
Max Simultaneous Targets 

Target /Keytrame Face Abcounk Puirwing Detection t : Target Keyfranie|race . .utoInit) Painting Sctection — -| 
x Use Exclusion Image 
Ww Optimized Reprojection ? j aeoreivin 
Keyl Estimation Auto Init Pointing Detection |Moye Detector tes 
-Keyframe ————_ ——"| 4 


Filter 
Buffer Size 
Nb of Same Id Min 
Nb of Same Id Consecutive 
Create Import... Display Nenegsthre 


Remove Image(s)... Freeze 


~ Freeze delay ~Keyframets) 


—Keyframe Option 
Use Debug Mace 
Use GPU 
~ Pointing Keyframe Move Threshold 
Vertical Flip Hori sntal Flip 


Rotate 


-Keyframe Info 
Pos: X 0,00 Y 0,00 Z 20,00 Win SIZé Loc 
® 3.14 ¥ 0.00 Z 06,00 Threshold Loc 


Reset Pose Display Axis Win Size Dec 


Threshold Dec 


© Trackball Mode x 0,43 x 0.50 
Trackball Speed Delete Zone Detect: Threshold 


Priority om DOW e Sesrch fives Siva 


Ty, 
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Create a planar object (as explained previously) with or without automatic initialization. 
Check the “Pointing Keyframe” check box in the sub-panel “Keyframe”. 


Switch to the “Pointing Detection” sub-panel and reproject the requested keyframe using the 
“Reprojection” button. 


Create and remove areas using the mouse over the rendering window. Created areas will be declared in the 
“Zone(s)” list. 


Specify priorities over defined areas using the “Priority” list. 


Once pointing detection areas are specified, you can adjust pointing detection thresholds and parameters in 
the “Tracking Manager” panel. 


Use the debug mode (“Use Pointing Detection” check box parameters in the sub-panel “Pointing 
detection”) to display the results. You will see a second window named “Current Rendered Image” with the 
debug data of the pointing detection. In this second window, you will have your areas in blue rectangles 
and when you put something in front of them, the area becomes red after few instants. Notice that green 
points indicate that the area is not occulted; yellow points indicate that there is a small but insufficient 
perturbation and blue points indicate a strong occultation. These blue points are then used to decide if there 
is an occultation. 


During the tracking, you can adjust pointing detection live parameters. Among these parameters, notice 
especially the parameter “Move Threshold” that can be used to disable the pointing detection engine 
when the object moves in front of the camera. 
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4.12 HOW TO MAKE THE TRACKING MORE ROBUST AND STABLE 


Notice that this functionality is available only with the Classic Tracking Mode. 


In week conditions, typically when using a low cost webcam, some options in the configuration file can be 
adjusted to obtain a more robust tracking. Four main solutions can be used together or separately: 


« Tracking Manager » Panel « Tracking Manager » Panel « Tracking Manager » Panel 


Parameters ---- @- @ —W——. i _ parameters ——————__________. Parameters 


) ) 


‘ Painting Detection | Move Detector process 


) |Keyframefs) | Pose Estimation | Auto Init |Poinpc Target Tracking | Interest Paints | Correlation |K 


Minimization Total Taraet To -Parameters 
¢ é Move Detector 
Tracking Method Performances Display Ve es teal 
Use Weighted Iteration Display Wireframe on Init Nb Frame Init 
peers Display Keyframe 
Use Weights on Correlati “a ES Ke i : Fixed Point Radius 
Display Coordinate Systern 
Use Weights on Borders —3 Min Nb Fixed Points 
Rerender 
4rea Percent Required 0.10 
Use GPU 
Use Ransac for KF Pose Estimation Max Features 1000 
Face Cull Angle 89.9 
ation 50 
Pick Nearest Facet 
7 ojecti esho 5,00 
puer Reprojection Threshold mit Reprojection Local Patch 


Use R. for PKF Pose Estimati oca 
Y Use Ransac for ‘ose Estimation Reprojection Thresho : ocal 
Iteration 50 A 


Front Clipping Plane Integ 


Inlier Reprojection Threshold 
Back Clipping Plane 


Recursive 
-Sub Image 
Recursive Tracking Frames 
Convergence Iteration 
v USE KLT Recursive Tracking 
Scanning Mode 


1. A RANSAC Filtering. In order to use this mode, the user should first turn on the parameters “Use Ransac 
For KF Estimation” and “Use Ransac For PKF Estimation” Using this functionality, the user should be 
able to proceed faster movements and the tracking should be more robust to track object with low texture 
information. To ensure a full stability, increase the “Iteration” values to 300. 


2. An optical flow solution (based on the KLT algorithm) to be robust when video stream is blurred. In order to 
use this mode, the parameter “Use KLT Recursive Tracking” should be activated. 


3. A special reprojection mode for planar objects. When activated, this functionality constructs more reliable 
keyframes to improve robustness, especially when the object is far according to the camera position or when 
the orientation change is important. To activate this mode, select the “Integral (2D)” or “Integral 
Interest Points (2)” modes in the “Reprojection” list. 


4. An adjustment of the pose estimation engine. Activate the “Robustness” mode in “Tracking Method” list, 
and check the “Use Weighted Iteration” button. 


5. A stabilization of the pose estimation engine using weights that are computed using recursive correlations by 
checking the “Use Weights on Correlations” box. 


a 


Filter the pose estimation process. Using the “Move Detector”, the object position will be fixed automatically 
when both the camera and the target are static. Notice that the parameter “Fixed Point Radius” can be 
reduced (e.g. to 1.0) in order to have a smother tracking when the object or the camera moves. 


Caution: Notice that all these functionalities are time consuming. 
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4.13 HOW TO TRACK IN HIGHER RESOLUTIONS 


In order to be able to recognize and track smaller objects in front of the camera, it’s possible to adapt the 
processing size and few other parameters. Notice that the user should have a good enough hardware 
configuration to ensure that the tracking rate stays stable. 


« Video Processing» Panel « Tracking Manager » Panel « Tracking Manager » Panel 


Import Calib,, , 
-Camera 
Optical center (x, y) 960 570 


Focal (x, ¥) 1900 1900 - ; 
a MEE S| Ha 


Faveay) 53.6116 


1920 Target Tracking | Interest Points Correlation |K » f Target Tracking [Interest Points Cortelation |, 
Distortion A @@@$@$J \A@__@$€_@€@€$?_______ Patt meters Parameters 
Radiesse Display Keyframe Interest Paints Display Correlation 
: Display Previous Frame Intere: 
Tangential (kL, k2) 
" i Re Max Number 300 
Activate distortion Window Size 
Quality 
Preprocessing 2 4 —— Threshold 
Alpha 
Processing s| 
Init Max Number 


v Skip odd oF even Frame 


Window Size 
Sub Image 


- Derive Size 
Use Sublmage 


Number of Regionis) 


1. Set the “Processing Size” to the chosen resolution (between 480x270 and 960x1080 when using HD 1080i). 

2. Increase the interest points number (“Max Number”) to 300 or more. 

3. Set the “Window Size” and “Derive Size” to 5 or 7. 

4. Set the “Max Disparity” value to 25x25 and the window Size to a value between 15 and 21 (take car with this 

parameter since it is greedy). 
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4.14 HOW TO REMOVE PARTS OF THE TARGET 


Notice that this functionality is available only with the Classical Tracking Mode. 


In order to avoid detection errors and to avoid non trackable areas on 2D or 3D targets, the GUI provides the 
user a simple tool to remove parts of the targets. This tool is available using the GUI’s “Scenario Manager” 
panel and the “Exclusion Areas” sub-panel. 


é a 4 " 
© D'Fusion Studio Computer Vision 


File View ? 


|Video Manager |Camera Calibration | Video Preprocessing Tracking Manager | Output Manager 


Inst Create 


Import... 


( Face Autotnit Pointing Detectior 


je —— 


Validate 


Ges 


Use exclusions for tracking 


Once a keyframe is created for a given object, you can remove some areas using the following steps: 


1. Open the “Exclusion Areas” sub-panel 
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Select the keyframe you want to modify. Once selected it will appear on the display window (left side). 
Use the mouse to draw a polygon (convex or not) on the top of the displayed keyframe. 
Validate the current exclusion contour using the “Validate” button. 


ee ed 


Delete a selected area (in case you need to) using the “Delete Area” button. 


Notice that these areas will act as exclusion areas for automatic initialization and for recognition. 


e If your keyframe is already a “Auto Init” keyframe (binary trees already registered), you will have to 
reconstruct information concerning automatic initialization. 


e If you want to use these exclusion areas during the tracking step as well, you will have to select the 
“Use Exclusion For Tracking” edit box. 
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4.15 HOW TO USE THE BLACK BOX CAPABILITY 


There are two ways of using the Black Box capabilities: from the D’Fusion Studio CV Pro interface (explained 
below) or using the Batch Generator for “Lite” tracking (explained in section 5.2). 


Below you will find a description of best practices and recommendations. 


Recommendations for a good recognition 
e use textured images 
e use dissimilar images 
e remove text zones with small font size from original images (apply uniform background color) 
e Separate the images in different groups: 
o it helps differentiating 2 images (that are similar), included into two different groups 


o you can chose to process only one group per (video) frame, ensuring you will preserve video 
rate. 


How to adjust tracking project for Mobile Target deployment 
e Define only one object if possible 
e Include all images for this object, organizing them into groups 
e Chose to process only one group per image 


e If there is no need to detect nor track more than one image at a time, please ensure there is only one 
object instance. 


e Please control the following parameter: in the Scenario Manager tab, set the field "Nb Max Interest Per 
Keyframe" to 100 or 150. 
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From the D’Fusion Studio CV Pro interface 


Inst Create 


Import... 


Remove 


Plane Clack Box 


Plane Size \y 10 h 


Keyframe 


v group_0 
t IM G_000. 2 


Remove Image(s)... 
Remove Image(s)... (6) 


Nb. Group Detections per Frame Nb. Group Detections per Frame 


Generate Classifi-cs Remove Classifiers 


Nb. Max Interest Pointe rer Frame 300 


Nb. Max Interest Points per Frame 
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Keyframe 


wv, group_0 


Ee IMG 0002.PG 


group 1 

+f IMG_0006.JPG 
+f IMG_0007.)PG 
+f IMG_0008.JPG 
+ IMG_0009.JPG 


Remove Image(s)... 


Nb. Group Detections per Frame 


Generate Classifiers Remove Classifiers 


Nb. Max Interest Points per Frame 150 


Figure 28 — The “auto-init” icon next to the images 


1. Open the video capture and calibrate your camera. 
2. Activate the “Lite” tracking mode. 
3. Create a target from the “Scenario Manager” panel and select “Plane Black Box” as target type. The 
following dialog pops up: 
Optimization 
Perfomances Rebustness 
Mobile devices (iPhone, ...) 
OK Cancel 
Figure 29 — Dialog box performances of your black box object 
Select the optimization level of your scenario according to your targeted device. This will automatically 
configure your object black box to ensure a reasonable processing time on different devices. For 
example, if your scenario is to be deployed on a mobile phone, you may want to favor good 
performances over detection robustness in order to maintain a good framerate. 
Select “Ok” when you are done. 
4. Import the images to recognize using the “Image(s)...” button. You can add multiple images at once 
using CTRL or SHIFT+select 
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Note: Each import creates a new group and adds the selected images to the group. There is no 
constraint on the number of images per group (i.e. you can have a group with 4 images and a group 
with 10 images) and the aspect ratio of the images of a group does not necessarily have to be the same 
(i.e. you can have a 4:3 image and a 6:9 image in the same group). 


Define the number of groups to process per frame. The number of groups to process per frame highly 
depends on your hardware (CPU) and will impact performances. 


Click on the “Generate classifiers” button to create the static equivalent of your targets. An “Auto Init” 
icon will appear next to the images for which a classifier has been successfully generated. 


When saving your scenario, the following files will be created next to your tracker.xml file: 


dictionnary.bin: this is the “black box”, required for each scenario with “Planar Black Box” targets (one 
file per scenario). The size of this file is always the same and does not depend on the number of images 
registered into your scenario. 


<targetName>_partitionTree_<groupNumber>.bin: contains the partition tree of the group 
<groupNumber>, required to identify the images registered into this group for the target 
<targetName>. 


<targetName>_signatures_<groupNumber>: contains the signatures of the images registered into the 
group <groupNumer>. 
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4.16 HOW TO CREATE DYNAMIC GROUPS WITH BLACK BOX 


This section explains how to create Dynamic Groups when using the Black Box features. 


1. Open the video capture and calibrate your camera. 
2. Activate the « Lite » mode. 
3. Create an object (« Scenario Manger » panel). 
4. Set the target type to « Plane Black Box ». The following dialog pops up: 
Wi] Classifiers Black-Box properties 2 Si 
timization 
Perfomances Robustness | 
Personal Computer 
OK Cancel 
Figure 30 - Dialog box performances of your black box object 
Select the optimization level of your scenario according to your targeted device. This will automatically 
configure your object black box to ensure a reasonable processing time on different devices. For 
example, if your scenario is to be deployed on personal computers, you may want to favor detection 
robustness over higher processing time. 
Select “Ok” when you are done. 
5. Click on the « Image(s)... » button to select your images. A group to contain the selected images is 
created automatically. 
Every time you select a collection of images through the « Image(s)... » button, a new group is 
automatically created to contain the selected images. 
6. Double-click on a group to change the group’s name. 
7. Define the number of detections to perform per video image. 
8. Click on the « Generate Classifiers » button to create the static equivalent of your targets. The « auto 
init > icon appears next to your image for which the classifiers have been generated. 
9. Right-click on a group's name to specify if it is to be saved as dynamic. 
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Remove 


Max Simultaneous Targets 1 Max Simultaneous Targets 1 


Target Target 

Type Plane Black Box Type Plane Black Box 
Plane Size w 10 Plane Size Ww 10 h 10 
Keyframe Keyframe 


A object 1 grot — " sete Y » objectt : 
ynamic Grou : ae 
L +4 000001. jp. as P L ooo ” Dynamic Group 


v object_1_group 1 v object 1_group_1 
L- 44. 900017,jpg L$. 900017,jpq 


Figure 31 — Black Box group contextual Figure 32 — “is dynamic” group icon 
menu 


10. Save your scenario (File > Save). The following dialog box allows you to choose the saving location of 
the « runtime » and the « authoring » scenarios, and the path to your dynamic groups: 
" . 


ai Sav 


~ Default architecture 


Tracking scenario directory 


E; \SaveBlackBoxDynamic 


tracker. xml 


Dynamic tracking scenario directory 


E:\SaveBlackBoxDynamic\dynamictracker \ 


Dynamic group{s) directory 


E: \SaveBlackBoxDynamic\dynamictracker \ 


| 


Figure 33 — Saving dialog box of a black box scenario with dynamic group 
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A default saving tree is proposed if the checkbox « Default architecture » is checked. In this case a 
“dynamic” folder named after your tracker is automatically created in your scenario folder. For example, 
if your tracker is named Tracker.xml, the folder is named “dynamicTracker”. This folder contains the « 
runtime » scenario (i.e. the one that will be loaded when launching your application) and a folder for 
each dynamic group. 


4.16.1 Examples 


4.16.1.1 Scenario with no dynamic groups 


The following example illustrates the files and folders that result from saving a D’Fusion Studio CV Pro scenario 
with no dynamic groups: 


db img 25/10/2011 11:32 Dossier de fichiers 

4 dictionnary.bin 20/05/2011 10:16 VLC media file (.bi... 4 966 Ko 
4 object_1_partitionTree_0.bin 25/10/2011 11:32 VLC media file (.bi... 17 Ko 
4 object_1_partitionTree_1.bin 25/10/2011 11:32 VLC media file (bi... 15 Ko 
4 object_1_signatures_0.bin 25/10/2011 11:32 VLC media file (.bi... 29 Ko 
& object_1_signatures_1.bin 25/10/2011 11:32 VLC media file (.bi... 23 Ko 
__, tracker.xml 25/10/2011 11:32 Fichier XML 22 Ko 


Figure 34 — Folder contents of a scenario saved with no dynamic groups 


4.16.1.2 Scenario with dynamic groups 


If the scenario contains dynamic groups, a dynamic folder named “dynamic<NameOfTracker>” is created when 
your scenario is saved from D’Fusion Studio CV Pro. 

This folder contains the “<NameOfTracker>.xml” file to use into you AR scenario (i.e. the runtime scenario), 
and the Lua function MLTPluginManager:startTracking(): 


MLTPluginManager:startTracking (™..\dynamic<NameOfTracker>\<NameOfTracker>.xml”, ...) 


,) dynamictracker 25/10/2011 11:32 Dossier de fichiers 

ab img 25/10/2011 11:32 Dossier de fichiers 

& dictionnary.bin 20/05/2011 10:16 VLC media file (bi... 4.966 Ko 
4 object_1_partitionTree_0.bin 25/10/2011 11:32 VLC media file (.bi... 17 Ko 
4 object_1_partitionTree_1.bin 25/10/2011 11:32 VLC media file (.bi... 15 Ko 
4 object_1_signatures_0.bin 25/10/2011 11:32 VLC media file (.bi... 29 Ko 
& object_1_signatures_1.bin 25/10/2011 11:32 VLC media file (.bi... 23 Ko 
__, tracker.xml 25/10/2011 11:32 Fichier XML 22 Ko 


Figure 35 — Scenario folder contents of a 
scenario saved with dynamic groups 
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ad img 24/10/2011 17:33 Dossier de fichiers 
r) object_l_group_1 24/10/2011 17:33 Dossier de fichiers 
4 dictionnary.bin 20/05/2011 10:16 VLC media file (.bi... 
4 object_1_partitionTree_0.bin 24/10/2011 17:33 VLC media file (.bi... 
4 object_1_signatures_0.bin 24/10/2011 17:33 VLC media file (.bi... 
__, tracker.xml 24/10/2011 17 Fichier XML 


Figure 36 — “dynamictracker” folder contents 
if 1 non-dynamic group and 1 dynamic group 


ab object_1_group_1 26/10/2011 13:28 Dossier de fichiers 
db object_1_group_2 26/10/2011 13:28 Dossier de fichiers 
4 dictionnary.bin 20/05/2011 10:16 VLC media file (.bi... 
__| tracker.xml 26/10/2011 13:28 Fichier XML 


Figure 37 — “dynamictracker” folder contents 
if 2 dynamic groups and no non-dynamic group 


a img 24/10/2011 17:33 Dossier de fichiers 
__, object_1_group_1.xml 24/10/2011 17:33 Fichier XML 

4 object_1_group_1_partitionTree_.bin 24/10/2011 17:33 VLC media file (.bi... 
4 object_1_group_1_signatures_.bin 24/10/2011 17:33 VLC media file (.bi... 


Figure 38 — dynamic group “object_1_group_1” 
folder contents 


In the “dynamic<NameOfTracker>” folder you can find: 


e The “<NameOfTracker>.xml” file: this is the main runtime tracking description. 


4 966 Ko 
17 Ko 
32 Ko 


21 Ko 


4 966 Ko 


21 Ko 


11/12/12 


e A“dictionnary.bin” file: this is the Black Box itself. It describes and enables the Black Box features. This file 


is identical for all projects, regardless of your project content. 


e For each non-dynamic group (i.e. to be loader/unloaded when the scenario is launched), a file named 


“<objectName>_partitionTree_<groupName>.bin”. 


e For each non-dynamic group (i.e. to be loader/unloaded when the scenario is launched), a file named 


“<objectName>_signatures_<groupName>.bin”. 


e = An“img” folder containing the images of non-dynamic groups, if non-dynamic groups have been defined. 


e For each dynamic group, a folder with the name of the group (e.g. “object_1_group_0” in the example 


above) containing: 
o A file named “<objectName>_partitionTree_.bin” 


o A file named “<objectName>_signatures_.bin” 


o A file named “<groupName>.xml”, which contains the configuration for the group. 


o An“img” folder which contains all the images in this group. 
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4.17 HOW TO CHOOSE IMAGES FOR BLACK BOX RECOGNITION 


The quality of the image recognition is dependent on the characteristics of the images to be recognized. 
Below is a listing of the key features for best results 


e Large surface textured objects 


e Avoid small surface textured 


e No symmetry or similarity allowed 
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Avoid object with text 
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4.18 HOW TO HELP THE RECOGNITION WITH SIMILAR OBJECTS 


Similar objects are difficult to recognize so you should take this into account when designing your application. 
Similarity can be global on images (same image, with only few variations) or local (same patterns repeated all 


Sg me | ime | lem 
bk ce bt 
ee) FF] Oe & 


— = | «7 i mn 
= FONCTION PANCREATIQUE 
Bs al 


g . F 4 
SURCHARGE PONDERALE | 
: . . . be 
fa rR 
i DIABETE DE TYPE 2 d 
POUR TOUT VOIR, vr 1 


CHANGEONS DE POINT DE VUE DIABETE DE TYPE 2 
OUR TOUT VOIR, 


L - CHANGEONS DE POINT DE VUE 
no nord L 


Figure 39 — An example of similar images 


In case you have too similar images to recognize, the first thing to do (when possible) is to remove parts of 
the images as explained in the previous section. 


Another possible solution to help the recognition is to activate the Auto Init -> Search Mode -> Best First in 
the Tracking Manager panel. In this case, the engine will match all objects and compare the results to decide 
the most relevant one. 
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4.19 HOW TO CREATE A LITE SCENARIO USING THE BASIC USER LEVEL 


The basic user level helps you define very standard “Lite” scenarios using the following steps: 


1. Configure and Open the video capture using the Video Manager panel. 


2. Configure the scenario using the Scenario Manager panel. 


a. 
b. 


c. 


Create an object. 
Specify the number of instances for this object (field instances). 


Import images using the Jmage(s)... button. Note that these images will be resized 
automatically to fit with the video capture size (usually width - or height depending on the 
aspect ratio - is resized to 250 pixels). 


Specify the number of images you want to gather in a group for the recognition step (/Vb. 
Images per Group edit box). If you want to recognize many images you can set this value to 
[5..10]. 


Specify the number of recognition to be processed for each frame (Wb. Detections per Frame 
edit box). Don’t change this parameter unless you need to speed up the recognition step. 


Then finally, generate the classifiers using the button Generate Classifiers. At this step, a box 
appears and asks you the required quality for these classifiers. The highest it is, the more 
robust the recognition will be; but at the cost of memory usage. 


3. Check the Recognition and the Tracking in the 7racking Manager panel. 


4. Then save your scenario. 
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5, ADVANCED USERS 


5.1 HOW TO CALIBRATE A CAMERA MANUALLY 


In some case, it is difficult to calibrate a camera because of the environment lightening. This section explains how 
to process the calibration quickly and especially the focal values. 


e The Calibration for the MarkerLess Tracking looks like the left panel for a HD/SDi Camera 


e fx, fyare the focal lengths in pixels (values we want to estimate), and cx, cy denotes the image center (in 
pixels) that can be set to image_size/2. 


e How to figure out the & & fy values ? 


Place a rectangular object in the front of the camera. 

Measure this object width and height (in inch, centimeters, ... or whatever unit you want) = w & h. 
Measure the distance between the optical center and the object (a). 

Measure the object width and height (in pixels) into the corresponding image (wp, Ap). 

Calculate & = d*wp/w & fy = d*hp/h. 

o Report these values into the GUI. 


00000 


e If you know your fovX and fovY (view angles), you can approximate the fx and fy values as well : 


o «6k = 960/(2*tan(fovxX/2)), and 
o fy = 540/((2*tan(fovY/2)). 


5.2 BATCH GENERATOR FOR LITE TRACKING 


The goal of the DfusionCV_Generator.exe tool is to automatically create the set of configuration files and 
statistical classifiers for the D’Fusion Studio CV Pro Lite application, from a given set of images or some 
configuration files (such as the tracker.xml file). It is useful when the tracking scenario has to be generated 
automatically for instance within an application generator workflow running on a server backend. 


There are three ways to use this tool. 


5.2.1 Complete scenario creation (with standard classifiers) 
D’Fusion CV accepts several image formats as input: jpg and also bmp (or a directory containing a set of images), 
and outputs standard configuration files. 


The tool creates objects (to be recognized and tracked) according to the different given images. Images are 
passed as arguments. Generated files are stored in this same folder as the images, it includes: tracker.xml, 
cameraCalib.xml, cameraConfig.xml and the images classifiers. 


If all the images have the same aspect ratio (width, height), only one object is created. Otherwise, the tool will 
generate as many objects as different ratios. The number of trackable objects at the same time depends of the 
number of images with different ratios. 


The images are also resized to have a size which optimizes the quality of the recognition and the tracking 
processes. Images are resized to have a width (or height) of 250px. New images are saved in the original folder 
in jpg format and they keep the same name with “.jpg” at the end. 


Example: 
Input image: "rabbit.jpg" 1920x1080 
Output image: "rabbit.jpg.jpg" 250x141 
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Images for a single object are reassembled in “classification” groups (stored in the file “object_1.0.bin”, 
“object_1.1.bin” ...). There are 4 keyframes per group. 


Generated camera calibration is the default one (stored in the “CameraCalib.xml” file): 
<MAXX>640</MAXX> 
<MAXY>480</MAXY> 
<PX>320.00000000</Px> 
<PY>240.00000000</PY> 
<FX>900.00000000</FX> 
<FY>900.00000000</FY> 
<A1>0.00000000</A1> 
<A2>0.00000000</A2> 
<P1>0.00000000</P1> 
<P2>0.00000000</P2> 
<DEDISTORTION>false</DEDISTORTION> 


A video capture file is also built (and stored in the file “CameraConfig.xml”) to open the camera indexed 0 with a 
640x480 resolution. 


e -q 0...5 (0 = 128KB ; 1 = 256KB ...) specifies the quality of the classifier that will be generated. The quality 
of a classifier influences the final size of the scenario and will therefore impact performances (memory 
usage). 


5.2.2 Complete scenario creation (with “Black Box” capabilities) 


To create a scenario using the “Black Box” capability, you have to add these arguments to the command line: 
e -b indicates that you wish to generate the classifier using the “Black Box” 


e -g <NB_IMAGES_PER_GROUP> allows you to specify the number of flyers you want to group under the 
same classifier (for example 200). 


e -d, if the -b parameter is set, indicates that the groups attached to the black box object are dynamic. The 
“runtime” scenario is created (see section 2.3 Key Concepts “Black Box Dynamic Groups”) 


e Other arguments are standard, it could be : 
o Images that you want to register (.bmp or jpg). 
o Directory where the images are stored (easier if you have a lot of images to classify). 
o An.xml configuration file if you just want to rebuild the classifiers. 


5.2.3 Scenario update using an existing configuration file (tracker.xml) 


DfusionCV_Generator.exe can be used to regenerate classifiers when images are changed in the scenario. In 
this case, the unique argument is the tracker.xml file that you want to use to regenerate classifiers. Note that you 
will have to remove old classifiers from the directory (.bin files) before regenerating the files. 


In practice, you can drag and drop your xml file on top of the .exe file to launch the process. When finished, you 
can see that new classifiers are generated. 
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6. TROUBLESHOOTING 


The D’Fusion Computer Vision software is highly configurable to deal with very different cases of tracking. As a 
consequence, mistakes when specifying parameters may introduce trouble during the real-time tracking: 

e Classifier version too old 

e Application failure 

e Initialization step failures 

e Tracking step failures 

e Important jitters (vibrations) when tracking the object pose 

e Low Performances 


To deal successfully with these issues, the user may refer to the following tables. 


Classifiers too | When launching a D’Fusion tool. 
Old 
Solutions Launch the application using the D’Fusion CV Player or the D’Fusion Studio CV Pro to 
regenerate classifiers automatically. 
Ensure that old versions of the classifiers are correctly removed from the scenario directory. 
Application When launching the D’Fusion CV Player it exits immediately. 
Failures 
Solutions Check the message displayed if any message box, or the warning outputs into the 
"warning. txt file. 
Ensure that video capture parameters are correctly defined according to the camera 
specifications. 
Application When launching D’Fusion Augmented Reality, the plug-in is not loaded correctly 
Failures 
Solutions Check the message displayed if any message box, or the warning outputs into the 
"warning. txt file. 
Ensure your antialising is not controled by the driver application (see the graphic card 
configuration panel). If it is the cases, use antialising 8x or 4x. 
When using multi-target capabilities, ensure the number of channel in mocap objects 
corresponds to the target number. 
Initialization The target cannot be found in the image and the application remains in the initialization step. 
Step Failures 
Solutions Increase the correlationWindowSize (“Window Size”) 
Increase the interestPointsMaxNumber (“Max Number") 
Increase the interestPointsInitMultiplier (Init Max Number’) 
Increase the correlationThreshold (“Threshold”) 
Ensure the 3D grid model had been defined with the correct orientation. Red facets 
must appear when displaying the keyframes. 
Tracking Step | The target is found but for a very short time only and the application goes back into the 
Failures initialization step. 
Solutions Increase the interestPointsMaxNumber (“Max Number”) 
Ensure that the “Use Ransac For KF Estimation” and the “Use Ransac For PKF 
Estimation” check boxes are checked. 
Enable the “KLT Recursive Tracking” with at least 10 Frames (see the “Recursive 
Tracking Frames” edit box). 
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Jitters The target is found but small vibrations appears when considering its pose 
Solutions Increase the interestPointsMaxNumber (“Max Number”) in case of low textured objects 
such as faces 
Decrease the interestPointsMaxNumber (“Max Number”) in case of hightly textured 
objects such as faces. 
Decrease the interestPointsQuality (“Quality”) in case of low textured objects such as 
faces. 
Increase the interestPointsQuality (“Quality”) in case of hightly textured objects such as 
faces. 
Decrease the correlationThreshold (“Threshold”) 
Ensure the 3D grid models fits with the real object geometry (for complex 3D geometries 
only). 
Choose the “Robustness” option in the “Tracking Method” list. 
Click the “Use Weighted Iteration” check box (this will cost CPU computation). 
Low The D’Fusion CV Player is too slow to track any object. 
Performances 
Solutions Ensure the computer fits with the minimum configuration. 
Reduce the maxDisparityU,V (“Max Disparity”) area 
Reduce the interestPointsMaxNumber (“Max Number”) 
Reduce the interestPointsInitMultiplier (‘Init Max Number”) (works for initialization 
step only) 
Reduce the proceeded image size (ProcessingHeight, ProcessingWidth) or reduce the 
captured image size (video_capture_width, video_capture_width) 
When using D’Fusion Computer Vision as a plugin, ensure the CPU parameter is set to 1. 
In some specific configuration, (especially when using D’Fusion @Home® or the multi-target 
capabilities), it's possible to improve tracking performances. You can use one of the following 
adjustments by editing the .x/7/ configuration file : 
e The pose estimation engine can be adapted (see the [MINMIZATION] section 
or the “pose estimation” sub-panel in the GUI): 
Parameter’s name value 
TrackingMethod 5 
Nblterations 15 
UseWeighIterations O (default value) 
e Incase using the recognition engine, it’s possible to search only one object at 
a time. To do so, set the parameter SearchMode that belongs to the section 
[AUTO_INITIALIZATION] to “ONE_DETECTION_BY_TRAME” or to 
“N_DETECTIONS_BY_TRAME”. 
e In case using the recognition engine, try to set the parameters as follow (see 
the [PLANAR_DETECTOR] sections): 
Parameter’s name value 
Max_Ransac_Iterations 150 ... 200 
Max_Keypoints_Number 200 ... 300 
Patch_size 16 
Tree_Number 8... 16 
View_Number 50 ... 80 
Min_Scalei,2 0.5... 0.7 
Max_Scale1,2 1.1... 1.5 
Sample_Number_For_Refining 500 ... 800 
Notice that changing these parameters can result in some detection errors of 
the recognition engine. 
e Take advantage of the move detector engine in order to save computation 
costs when both the object and the camera are static. 
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7. REQUIREMENTS 


7.1 MINIMUM SYSTEM REQUIREMENTS 


Classical Tracking Mode: 


e¢ PC compatible with Intel Pentium or AMD class processor 
e 1 GHz processor 

e 512 GB system memory 

e USB 1.0/2.0 slot available 

e Windows 2000/XP/Vista/Seven compatible 


Lite Tracking Mode: 


e Flash Platforms, IPhone, Android OS ... 
e 0.5.. 1 GHz processor 


7.2 NATURAL FEATURES TRACKING REQUIREMENTS 


This section reminds the basics requirements concerning the tracking and the recognition. 


Requirements (1) 


m™ Camera requirements 
@ Avoid optical blur as much as possible 


m Avoid dynamic focal changes 
m Camera supported : WebCamera, UEye, HD camera... 
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Requirements (2) 


@ Object requirements 
m Large Surface Textured objects 


= = 


= No Symmetry or similarity allowed 
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Requirements (3) 


@ Object requirements 
@ Avoid objects with text only 


m Prefer textures to simple shapes 


ae 


L 
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Requirements (4) 


m@ Object requirements 
m Avoid Glossy and mirror surfaces 


TOTAL IMMERSION Beyond Reality 


Requirements (5 


m™ Recognition (Initialization) requirements 
= Simple 
= For textured 3D objects 
= For 2D objects 
@ with big text 
w with more longer than larger size 
= Auto-Initialization 
= Available only with 2D objects 
= Better with more or less squared images 


HE WE OsTous au numérique! ae 
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m Up to 15 images for an @Home application (require advanced 
user to set up a 15 images scenario) 


m= Up to 60 images on a kiosk (require advanced user to set up 
a 60 images scenario) 
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Requirements (5) 


= If you have several objects to track in your scenario 
m Avoid similarities between objects or images 


= If you want to track several objects at the same time, we recommend a maximum 
of 


m™ 2 images for an @Home application 
m= 3-4 images for a Kiosk application 


TOTAL IMMERSION Beyond Reality 


= Minimum object size in the live video stream 


m In Recognition mode (Initialization) 
= Minimum object size = % of the full image 
= You must put the image close to the camera 
m= In Tracking mode 
= Minimum object size : 1/16 of the full image 


= Once the image is detected, if you don’t loose the tracking, you can move the image 
away 


If you loose the tracking far away from the camera, put your image close to the camera to 
reinitialize it 


Recommended object size for recognition Possible object size during tracking 
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Requirements (8) 


® Pointing Detection requirements 
@ Available only with 2D objects 
m Avoid non textured areas 
= Avoid too small areas 
m Avoid non rigid flyers 


Non Rigid objects Non Textured Area 
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